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EXECUTIVE  SUMMARY 


<  A 

the  objective  of  this  study  was  to  conduct  an  evaluation  of  the  com¬ 
puter  assisted  instruction  (CAI)  segments  of  the  TOW  Field  Test  Set  (TFTS) 
Course  and  the  HAWK  Continuous  Wave  Radar  Repair  Course  conducted  at  MMC&S. 
Computer  Assisted  Instruction  for  both  courses  was  provided  via  the  PLATO  CAI 
system.  PLATO  was  brought  into  the  TFTS  course  because  it  was  anticipated  that 
very  few  TOW  Field  Test  Sets  would  be  available  at  the  school  for  training. 
PLATO  instructional  segments,  which  could  be  better  characterized  as  "simula¬ 
tion",  were  interspersed  throughout  the  TFTS  course.  All  TFTS  students 
received  PLATO  segments  as  an  integral  part  of  their  instruction.  There  was  no 
"non-PLATO"  control  group  available. 

The  HAWK  course  was  not  constrained  by  equipment  shortages  in  the  way 
the  TFTS  course  was.  The  PLATO  segments  of  the  HAWK  course  were  very  small  in 
comparison  to  the  length  of  the  overall  course.  Therefore,  with  respect  to  the 
HAWK  course  evaluation,  it  was  possible  to  have  a  control  (non-PLATO)  group  and 
v  an  experimental  (PLATO)  group  for  comparison  purposes. 

/Data  analysis  for  both  the  TFTS  and  the  HAWK  evaluation  was  organized 
around  the  questions  of  effectiveness,  efficiency,  user  acceptance,  quality  of 
implementation,  generalizability,  system  reliability,  and  cost  extrapolation. 

'The  results  are  that,  for  the  TFTS  course,  the  PLATO  simulations  were 
effective,  efficient,  accepted  by  students  and  instructors,  reliable,  and 
potentially  cost-effective.  The  HAWK  results  are  that  the  PLATO  simulations 
are  equally  effective  (test  results)  or  less  effective  (subjective  results),  of 
questionable  efficiency,  unacceptable  to  students  and  instructors,  reliable, 
and  less  likely  cost-effective. 
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INTRODUCTION 


The  purpose  of  the  study  was  to  conduct  an  evaluation  of  the  com¬ 
puter  assisted  instruction  (CAI)  segments  of  the  TOW  Field  Test  Set  Course 
(hereafter  referred  to  as  the  TFTS  course)  and  the  HAWK  Continuous  Wave  Radar 
Repair  Course  (hereafter  referred  to  as  the  HAWK  Course).  This  evaluation 
was  based  on  a  previously  developed  test  and  evaluation  plan*. 


Implementation  of  the  CAI  Simulations 


The  courseware  that  was  evaluated  was  implemented  on  the  Control 
Data  PLATO  CAI  system.  This  courseware  was  available  in  two  forms:  (1)  a 
time-sharing  supported  system,  and  (2)  a  stand-alone,  microcomputer  system. 
Materials  on  both  systems  were  very  similar  with  only  a  few  differences  due 
to  the  advantages  allowed  by  each  method  of  delivery. 

In  both  the  TFTS  and  the  HAWK  course,  the  style  of  use  and  the 
style  of  courseware  were  atypical  of  the  usual  CAI  implementation.  For 
example,  the  CAI  lessons  were  taken  simultaneously  by  two  students.  One  stu¬ 
dent  read  through  the  technical  manual  (TM)  step-by-step  while  the  other 
student  performed  the  actions  called  for,  by  touching  the  screen  or  typing  an 
answer. 


The  CAI  lessons  were  unusual  too,  although  they  appeared  to  be  very 
appropriate  for  their  use.  These  CAI  materials  could  more  properly  be  called 


*  Francis,  L.  D.,  and  Levy,  G.  W.,  A  Test  and  Evaluation  Plan  for  the  CAI 
Segments  of  the  TOW  Field  Test  Set  and  the  HAWK  CW  Radar  Repair  Courses, 
Battel  1  e-Columbus,  1981. 
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CAI  simulations  than  CAI  lessons.  That  is,  the  simulations  had  no  instruc¬ 
tional  objective,  per  se ,  other  than  the  overall  instructional  objective  of 
the  module.  The  "objective"  of  each  lesson  was  to  reproduce  as  closely  as 
possible  the  actions  of  the  actual  test  or  operating  hardware,  and  to  provide 
feedback  (much  like  an  instructor  would)  to  the  student  actions.  Because  of 
the  design  objectives,  enhanced  achievement  scores  and  retention  were  not 
expected.  Some  increase  in  student  motivation/attitudes  was  anticipated. 


A  Brief  Description  of  the  PLATO  System 


The  PLATO  computer-based  instruction  system  is  a  modern,  sophisti¬ 
cated  delivery  system.  As  used  for  HAWK  and  TFTS  training,  the  system 
consisted  of  a  high-resolution  (512  x  512  dot)  monochrome  CRT  equipped  with  a 
16  x  16  touch  sensitive  panel.  The  keyboard  contained  about  a  dozen  extra 
keys  that  facilitated  the  instructional  process  and  that  were  labeled  for 
instructional  uses  (HELP,  LAB,  BACK,  etc.).  An  internal  microprocessor  con¬ 
trolled  the  system.  Some  of  the  PLATO  terminals  at  MMCS  were  connected  to  a 
large  mainframe  control  computer  by  telephone  lines.  Other  terminals  (called 
micro-PLATO  terminals)  got  their  data  only  from  connected  floppy  disks. 
Although  centrally-connected  terminals  are  needed  to  create  courseware,  once 
created,  the  courseware  can  be  delivered  more  inexpensively  via  stand-alone, 
disk-based  terminal  systems.  With  proper  attachments  micro-PLATO  terminals 
can  use  central  PLATO  services.  This  evaluation  focused  on  the  most  effi¬ 
cient,  minimum  cost  configuration  that  could  be  used  for  long-term  courseware 
delivery  and  notes  only  briefly  alternate  configurations. 


Lesson  Topics 

The  lesson  topics  for  the  TFTS  course  were: 

Controller  DVM  Alignment 
Controller  Counter  A1  Alignment 
Controller  Timer  A7  Alignment 
Controller  Timer  A6  Alignment 

Functional  Analysis  and  Alignment  of  the  Power  Source  Unit 

Functional  Analysis  and  Alignment  of  the  Power  Supply 
Modulator/Test  Adapter 

TOW  Weapon  System  Verification  Test  (6  lessons) 

2  new  lessons  on  Troubleshooting  written  by  MMCS 
course  staff  (not  yet  in  general  use) 

The  lesson  topics  for  the  HAWK  course  were: 


Repair  Antenna  Positioning  Circuits 
Repair  Horizontal  Amplifier  ^.nd  Power  S  'pply 


The  TFTS  Simulations  (15  -  17  hours)  were  placed  throughout  tne  three 
week  course  in  essentially  an  even  distribution.  For  nearly  every  exercise 
where  a  computer  simulation  could  be  predicted  to  offer  a  significant  advantage, 
that  simulation  has  been  created. 

The  HAWK  simulation  (1  -  1-1/2  hours)  was  located  past  the  half-way 
point  in  the  course.  It  was  a  very  saml 1  part  of  a  relatively  long  course 
(27  -  45  weeks). 


Description  of  TFTS  and  HAWK  Radar  Equipment 


The  TOW  Field  Test  Set  consists  of  a  group  of  about  a  dozen  devices 
that  are  used  to  check  the  readiness  of  and  troubleshoot  problems  for  the  TOW 
missile  and  its  firing  system.  Most  of  the  checks  made  are  electronic, 
although  there  are  some  optical  and  mechanical  tests  as  well.  The  test  instru¬ 
ments  are  relatively  large  (compared  to  a  simple  test  instrument  like  a  multi¬ 
meter)  and  rather  complex  in  terms  of  the  numbers  of  knobs,  gauges,  and  jacks. 
The  individual  devices  are  packed  in  large  trunks  wnen  moved. 

The  Improved  HAWK  Radar  Set  is  a  large  piece  of  equipment  mounted  on 
a  medium-sized  rubber-tired  trailer.  A  tower  supports  the  revolving  radar 
antenna  and  a  side  panel  opens  to  become  a  control  panel.  Troubleshooting 
typically  involves  isolating  a  printed  circuit  board  and  then  a  circuit  element 
on  that  board. 


EVALUATION  ISSUES 


Overall  Evaluation  Questions 


The  evaluation  attempted  to  look  at  the  general  questions,  "Were  the 
HAWK  and  TFTS  CAI  simulations  implemented  well  enough  to  meet  their  intended 
objectives?"  and  "Can  these  results  be  generalized  to  other  sites  and  situa¬ 
tions?"  This  was  to  be  done  by  examining  specific  issues  about  effectiveness, 
efficiency,  cost,  reliability,  and  user  acceptance.  The  extent  of  general iz- 
ability  is  an  important,  but  difficult  question  to  answer  w'th  certainty. 

The  results  of  any  single  study  have  low  general izabi 1 i ty  until  they  are 
combined  with  and  compared  to  other  studies  with  similar  objectives  and  para¬ 
meters.  Even  then,  generalization  is  limited  to  other  situations  where  the 
environment,  objectives,  and  situation  are  similar  to  those  studies  from 
which  the  generalizations  are  made.  Additionally,  all  evaluations  have  one 
further  limitation:  the  quality  of  the  implementation.  If  the  quality  of 
courseware  and  the  associated  instructor-led  teaching  is  high,  it  is  possible 
for  participants  (especially  instructors  or  school  administrators)  to  validly 
make  subjective  estimates  about,  for  example,  how  widely  applicable  this  CAI 
simulation  technique  is.  However,  if  the  quality  of  the  implementation  is 
low,  it  is  very  hard  to  believe  that  the  measures  of  user  acceptance, 
effectiveness,  etc.,  would  not  be  so  low  as  to  affect  any  estimate  about  how 
generally  useful  these  simulations  are.  Note  that  a  high  quality  implementa¬ 
tion  may  still  show  that  this  technique  could  be  applied  successfully  to  only 
a  few  other  courses.  Thus,  the  general izabil i ty  of  the  results  from  this 
evaluation  are  limited  by  the  quality  of  the  imolementation,  and  no  attempt 
to  draw  conclusions  and  recommendations  were  to  be  made  unless  the  implemen¬ 
tation  was  first  shown  to  have  been  satisfactory. 


Specific  Evaluation  Issues 


In  order  to  collect  the  data  to  answer  the  relevant  issues,  seven 
types  of  questions  were  asked. 

1.  Effectiveness.  Normally  this  question  would  be 
"Do  the  materials  teach?",  but  because  of  the 
objectives  of  these  simulations,  two  appropriate 
questions  for  both  courses  were  considered  to 
be:  "Was  the  simulation  fidelity  high?"  and 
"Did  transfer  occur?";  for  the  TOW  course,  "Was 
equipment  damage  reduced?";  and,  for  the  HAWK 
course,  "Was  there  a  significant  difference  in 
scores  and  grades  when  portions  of  the  course  are 
taught  via  simulation?" 

2.  Efficiency.  This  question  had  two  meanings  in 
the  context  of  these  courses.  First,  "When 
engaged  with  the  CAI  system,  was  the  student's 
time  well  used?";  second,  "When  CAI  systems  were 
introduced  for  practical  exercises,  was  efficient 


use  made  of  the  student's  availaole  time?" 


3.  User  Acceptance.  This  issue  was  raised  among 
two  groups:  students  and  instructors. 

In  various  formats,  the  essential  question  was, 

"If  you  had  a  choice,  would  you  try  to  use  CAI 
simulations  again?" 

4.  Re] iabi 1 i ty.  This  information  was  gatnered 
separately'  for  each  type  of  delivery  system. 

It  was  to  be  measured  in  a  quantitative,  oojec- 
tive  way  by  calculating  hardware  reliability  factors 
such  as  mean  time  oetween  failure  and  also  by 
subjective  estimates  of  the  comparative 
reliability.  Courseware  reliaoility  was  considerec 
as  an  integral  part  of  the  implementation  quality 
issue  (see  above) . 

5.  Cost.  Cost  extrapolation  was  to  answer  the 
question,  "How  much  would  it  cost  if  cnese  CAI 
simulations  were  permanently  installed  as  an 
on-going  part  of  these  courses?" 

The  issues  of  general izabi 1 i ty  and  quality  of  implementation  were 
treated  equally  with  the  five  issues  aoove,  but  have  been  discussed  pre¬ 
viously.  The  latter  issue  was  measured  both  by  Questioning  users  and  by 
analysis  of  logged  data  about  student  problems. 


Overall  Evaluation  Summary 


Looking  at  evaluation  only  from  an  "issues"  point  of  view  tends  to 
obscure  tne  reasons  and  objectives  for  the  original  decision  to  implement  the 
PLATO  system  in  tne  courses  chosen.  Therefore,  these  objectives  are  reviewed 
here. 


TOW  Field  Test  Set  Course 


PLATO  was  drought  into  the  TFTS  course  because  it  was  anticipated 
that  very  few  TOW  Field  Test  Sets  (TFTSs)  would  be  available  at  the  school 
for  training.  The  reasons  for  the  addition  of  PLATO-simulated  test  compon¬ 
ents  were  many: 

•  During  practical  exercise  (PE)  time,  four  students 
are  typically  paired  with  one  instructor.  It  is 
possible  for  only  two  students  to  work  at  a  piece 
of  equipment  under  the  instructor's  supervision. 

The  other  half  of  the  time  a  student  must  be 
occupied  with  self-study.  In  such  a  case,  the 
student's  time  in  PE  (particularly  PEI,  hands-on 
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training)  is  not  efficiently  used.  Taking 
advantage  of  this  time  for  PLATO  simulations  would 
keep  the  student  stimulated  and  motivated  and 
would  provide  nearly  double  the  "hands-on  time" 

(if  the  simulations  were  effective). 

•  furthermore ,  it  sometimes  happens  that  the  course 
starts  before  all  tne  TFTS  equipment  has  arrived 
and  been  readied.  In  that  case,  a  PLATO  simula¬ 
tion  could  provide  the  only  experience  a  student 
gets  before  having  to  work  in  the  field  with  a 
piece  of  equipment. 

•  Also  because  of  equipment  shortages,  the  avail¬ 
ability  of  spare  parts  or  replacement  items  was 
expected  to  be  low.  Therefore,  equipment  damage 
caused  by  students  who  misconnect  or  improperly 
control  the  test  equipment  could  have  a  major, 
long-term  effect  on  the  ability  of  the  school  to 
teach  the  course.  The  threat  was  very  real  in 
tnat  there  are  several  excellent  opportunities  for 
misadjusted  controls  to  damage  very  expensive 
equipment.  The  DLAT0  simulations  were  programmed 
to  catch  these  cases  and  provide  unr'orgetable 
feedback  about  the  consequences  of  such  actions 

on  the  actual  TFTS. 

•  Although  not  their  direct  responsibility,  school 
officials  looking  at  an  overall  Army  need  for 
training  realized  that,  for  those  portions  of  the 
course  with  PLATO  simulations,  it  might  be 
possible  to  have  training  exportable  to  anyone 
who  nas  a  TrS  arc  access  to  a  DLAT0  terminal  and 
the  TFTS  courseware. 

The  TFTS  course  had  no  comparison  group  for  a  CAI  vs.  non-CAI  study. 
Overall,  therefore,  its  evaluation  asked,  "What  does  having  PLATO  simulations 
available  add  to  this  course?" 


HAWK  Continuous  Wave  Padar  Repair  Course 


The  HAWK  course  had  a  comparison  group  (CAI  vs.  non-CAI)  available, 
but  was  not  constrained  by  equipment  shortages  in  the  way  the  TFTS  course 
was.  Therefore,  the  questions  asked  were:  "What  is  the  effect  of  replacing 
some  hands-on  equipment  experience  (which  has  little  feedback)  with  some  com¬ 
puter  simulated  experience  (with  feedback)?"  "Does  the  CAI  help  bridge  the 
gap  between  theory  classes  and  PE  or  does  it  enhance  motivation  (since  this  is 
a  relatively  long  course)?" 
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EVALUATION  DESIGN 


This  section  of  the  report  presents  the  evaluation  design  for  the 
TFTS  and  HAWK  courses.  These  designs  were  cased  on  discussions  at  MMCS  and 
a  thorough  exploration  of  alternatives.  They  appeared  to  be  the  most  feasible 
from  a  technical,  cost  and  schedule  viewpoint. 


TFTS  Evaluation 


The  TFTS  evaluation  was  severely  constrained  by  the  absence  of  any 
control  group.  There  was  no  comparable  group  of  students  tnat  was  taught 
the  TOW  Field  Test  Set  course  without  tne  use  of  PLATO.  "The  evaluation  of 
effectiveness,  therefore,  focused  on  whether  an  acceptable  level  of  perform¬ 
ance  was  achieved  by  an  acceptable  percentage  of  students.  The  evaluation  of 
efficiency  focused  on  two  questions:  whether  students  were  acceptably  trained 
using  fewer  equipment  end  items  per  student  than  in  traditional  courses  (i.e., 
four  students  per  equipment  end  item),  and  whecner  better  use  was  made  of  the 
students'  time  during  P.E.s. 

In  general,  the  evaluation  consisted  of  subjective  comparisons  (by 
students  and  instructors)  of  the  effectiveness,  efficiency,  acceptance,  and 
general izabi 1 ity  of  PLATO  simulations  compared  with  the  use  of  real  equipment. 


HAWK  Evaluation 


The  HAWK  effectiveness  evaluation  was  based  on  test  scores  for  the 
lesson  using  PLATO,  comparing  those  students  who  used  PLATO  wi tn  those  who 
did  not.  In  the  evaluation  design,  classes  would  be  randomly  assigned  to 
PLATO  and  non-PLATO  conditions  until  the  required  number  of  students  was 
obtained. 

Another  part  of  the  evaluation  consisted  of  subjective  comparisons 
(by  students  and  instructors  in  PLATO  classes)  of  the  effectiveness, 
efficiency,  acceptance,  and  general izabi 1 ity  of  PLATO  simulations  compared 
with  the  use  of  real  equipment. 

Since  the  amount  of  PLATO  use  could  vary  widely  in  the  HAWK 
classes,  a  question  on  amount  of  use  was  given  to  students,  for  possible 
use  as  a  control  variable  in  the  statistical  analyses. 


Sample  Size 


Minimum  sample  size  for  a  study  were  determined  based  on  three 
statistical  considerations:  (1)  the  desired  precision  of  estimation  or  the 


desired  amount  of  difference  that  is  to  be  reliably  detected,  (2)  the  desired 
confidence  level  for  the  study,  and  (3)  the  population  variance.  Desired 
levels  for  the  first  two  items  were  established.  The  third  item,  the  popula¬ 
tion  variance,  was  estimated  from  previous  experience.  Nonstatistical  consid¬ 
erations,  such  as  cost  of  additional  sampling  units  or  difficulty  in  obtaining 
additional  units,  also  figured  into  the  determination  of  sample  size.  Samples 
larger  than  the  minimum  were  requested  to  counteract  any  attrition  due  to  drop¬ 
outs,  missing  data,  etc. 


Sample  Size  -  TFTS 


The  minimum  sample  size  required  for  the  TFTS  evaluation  was  based 
on  the  desired  precision  of  estimation  for  the  student  subjective  judgments 
(since  there  was  no  control  group).  As  most  of  the  questions  are  based  on  a 
5-point  rating  scale,  this  type  of  scale  was  used  as  the  basis  for  calculating 
the  required  sample  size.  It  was  first  assumed  that  the  desired  precision  of 
estimation  is  just  under  one-half  of  a  scale  step  (i.e.,  .49  units).  That  is, 
if  the  mean  judgment  for  a  particular  auestion  is  2.5,  it  would  be  desirable 
to  have  a  large  enough  sample,  so  that  we  could  state  that  the  population  mean 
is  greater  than  2.0  and  is  less  than  3.0  with  a  specified  level  of  confidence. 
The  usual  level  of  confidence  selected  is  95  percent.  That  is,  we  would  make 
an  error  only  5  times  in  100  replications  of  the  study.  Other  levels  of 
confidence,  such  as  90  percent  or  99  percent,  are  sometimes  used. 


where: 


The  required  minimum  sample  size  ( N )  was  calculated  from: 

N  =  n  1^/  2 

N  0  '  l -a/2 '  7 

\  =  minimum  samcle  s^ce 

z~  -  pooulation  variance 

a  =  1  -  the  confidence  :eve"  ,3  =  1  - . 95  =  .05) 

^l-a/2  *  abcissa  of  the  unit  norma’  curve  corresponding 
to  l-ct/2  (I_s75  «  ’.96) 

w  =  width  of  the  conf*aerce  'otjrval  (w  *  t  .49  *  .98) 


Previous  experience  with  5-point  scales  has  shown  that  the  sample 
variance  (used  as  an  estimate  of  the  population  variance)  is  likely  to  lie 
between  1.00  and  1.96.  (This  latter  number  is  only  coincidently  the  same  as 
the  value  for  Z . 975 ) .  If  a2  =  1.96,  then  the  required  minimum  sample  size  was 
approximately  31_  students.  Actual  sample  size  attained  in  the  study  was  34 
students. 


The  same  considerations  apply  to  the  number  of  instructors  and 
school  staff  required  to  provide  reasonably  precise  estimates  of  their  judg¬ 
ments.  Since  it  is  unlikely  that  there  would  be  as  many  as  31  different  TFTS 
instructors,  we  simply  had  to  accept  as  many  as  we  could  get.  (Because  of 
differences  in  PLATO  implementation  in  the  TFTS  and  HAWK  courses,  and  the 
different  experiences  of  the  instructors,  the  data  from  TFTS  and  HAWK 
instructors  were  not  combined.) 


8 


Sample  Size  -  HAWK 


The  minimum  sample  size  required  for  the  HAWK  evaluation  was  based 
on  the  amount  of  difference  in  mean  test  lesson  scores  between  PLATO  and  non- 
PLATO  classes  that  is  to  be  reliably  detected.  It  was  assumed  that  a  differ¬ 
ence  of  10  percentage  points  or  more  between  the  means  was  important  to 
detect.  Further,  it  was  assumed  that  a  difference  this  large  or  larger  should 
be  detected  90  percent  of  the  time  (90  out  of  100  replications  of  the  study). 
Finally,  the  level  of  confidence  was  set  at  95  percent. 


from: 


The  required  minimum  size  for  each  group  was  calculated 


N  -  2a2  (Za/2  ♦  Z£)2/d2 


where: 


=  minimum  sample  size  in  one  group 
a  -  estimated  population  variance 

a  =  1-tne  confidence  level,  or  tne  probability  of  rejecting 
the  nuli  hypothesis  when  it  is  true  (a  =  1-.95  -  .05) 

3  =  the  probability  of  accepting  tne  null  hypothesis  when 

it  is  false  (3  -  1-.S0  =  .10) 

Z  =  aocissa  of  tne  unit  normal  curve  corresponding  to  the 
subscripted  area 

d  =  difference  between  independent  means  (10  percent) 


It  is  estimated  that  the  sample  variance  is  likely  to  be  100  percen- 
age  points.  If  cr  =  100,  a  =  .05,  3  =  .10,  and  d  =  10,  then  the  required 
minimum  sample  size  in  each  group  was  approximately  2J_  students.  Actual 
sample  size  attained  in  tnis  study  was  35  students  in  the  experimental  group 
and  25  students  in  the  control  group. 


Evaluation  Variables 


The  evaluation  variables  (i.e.,  effectiveness,  efficiency,  user 
acceptance,  quality  of  implementation,  general izabil ity,  reliability,  and 
cost)  are  presented  in  Appendix  A,  Tables  1  through  9,  together  with  their 
metrics,  rationale  for  use,  and  data  sources. 


DATA  COLLECTION  AND  ANALYSIS 


Data  Collection 


Data  was  collected  on  the  TFTS  and  HAWK  courses  in  accordance  with 
the  data  collection  plan. 

1.  For  each  course,  the  school  designated  a 
person  who  was  responsible  for  gathering, 
making  minor  checks  upon,  and  storage  of  the 
course  data  during  the  evaluation  period. 

2.  HAWK  course  instructors  chose  five  test  items 
that  clearly  and  primarily  reflected  knowledge 
gained  during  the  portion  of  the  block  that 
was  taught  via  PLATO. 

3.  Instructors  who  used  PLATO  were  briefed  on: 

•  Usage  logs 

•  Administration  of  guestionnai res 

•  Problem  logs 

•  Recording  of  unusual  incidents  -  weekly 

4.  Copies  of  problem  and  usage  logs  were  reproduced. 

5.  Lists  of  terminal  number,  terminal  type,  and 
terminal  location  were  prepared. 

6.  HAWK  student  data  (common  subjects  scores,  and 
scores  on  the  relevant  test  items)  for  both  PLATO 
and  non-PLATO  classes  were  obtained. 

7.  The  evaluators  screened  end-of-course  evalua¬ 
tions  for  PLATO-related  comments. 

8.  At  the  end  of  the  PLATO  instruction,  the 
instructor  or  someone  designated  by  the  school : 

•  Verified  the  accuracy  and  completeness 

of  the  usage  and  problem  logs  and  checked 
that  back-in-service  time/data  were 
recorded. 

•  Recorded  any  anecdotes  about  class  events 
that  were  related  to  the  CAI  lessons. 

•  Transcribed  common  subjects  course  scores  to 
student  data  form  (HAWK  only). 

•  Distributed  and  collected  the  PLATO  end-of- 
course  evaluation. 

9.  Battelle  evaluators: 

•  Gathered  PLATO  cost  data  from  the  CDC  sales 
representative  and  reviewed  budget  completeness. 


[•] 


•  Gathered  TFTS  acquisition  and  repair  costs 
from  the  school. 

The  only  staff  questionnaires  received  for  both  the  TFTS  and  HAWK 
courses  were  from  the  course  instructors.  Other  school  administrative 
personnel  did  not  complete  these  forms. 


Data  Analysis 


Data  analysis  for  both  TFTS  and  HAWK  evaluations  was  organized 
around  the  questions  of  effectiveness,  efficiency,  user  acceptance,  quality 
of  implementation,  general izabil ity,  system  reliability,  and  cost  extrapola¬ 
tions.  Reliability  and  cost  data  from  the  TFTS  and  HAWK  courses  were  pooled. 
The  other  data  were  analyzed  separately  for  the  TFTS  and  HAWK  course  evalua¬ 
tions. 


Analysis  of  the  subjective  questions  involved  obtaining  frequency 
distributions  for  each  question.  Differences  between  students  and  instruc¬ 
tors,  and  between  TFTS  and  HAWK  courses  were  examined. 

The  HAWK  effectiveness  evaluation  involved  a  comparison  of  the 
means  on  the  test  lesson  for  the  PLATO  and  non-PLATO  classes.  Scores  on  the 
common  course  subjects  portion  of  the  course  and  the  estimated  time  spent 
were  used  to  interpret  the  effectiveness  results. 
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RESULTS 


This  section  reports  the  results  for  the  TFTS  and  HAWK  course  eval¬ 
uations  with  respect  to  effectiveness,  efficiency,  user  acceptance,  quality 
of  implementation,  general izabi 1 i ty ,  reliability,  and  cost. 


The  evaluation  will  be  discussed  issue  by  issue,  treating  all  eval¬ 
uation  items  for  one  issue  before  beginning  a  new  issue.  The  tables  found  at 
the  end  of  each  issue  discussion  will  direct  the  reader  to  the  specific  ques¬ 
tions  asked  on  each  questionnaire.  The  questionnaires  will  be  found  in 
Appendices  8-E;  the  frequency  of  each  response  has  been  entered  onto  the 
questionnaire  form.  The  discussion  of  the  issues  will  generally  note  only 
trends  or  consensuses,  leaving  the  details  of  the  response  profile  to  be  found 
in  the  appendices. 


Effectiveness 


HAWK  Effectiveness 


The  HAWK  effectiveness  evaluation  was  based  on  derived  5-item  subtest 
scores  related  to  the  PLATO  lessons,  comparing  those  students  who  used  PLATO 
with  those  who  did  not.  The  observed  mean  for  the  experimental  (PLATO)  group 
on  this  test  was  93"  (N=35),  which  was  slightly  worse  than  the  control  (non- 
PLATO)  group  mean  of  97"  (N=25).  With  respect  to  the  common  subjects  portion 
of  the  course  and  the  time  required  for  completion  of  the  HAWK  course,  the 
experimental  group  took  less  time  but  did  not  perform  as  well  when  compared  to 
the  control  group.  While  the  evaluation  plan  considered  the  use  of  covariance 
analysis  to  compare  the  experimental  and  control  group  on  the  five  item  effec¬ 
tiveness  measure,  this  was  not  done  because  the  distribution  of  scores  on  the 
five  item  test  did  not  lend  itself  to  covariance  analysis.  As  is  obvious  from 
the  means  achieved  (i.e.,  97%  vs.  93%),  the  scores  on  the  effectiveness  measure 
were  consistently  high  across  both  the  control  and  the  experimental  groups.  It 
is  difficult,  therefore,  to  establish  any  important  or  practical  difference 
between  the  two  groups  in  terms  of  the  five  item  test.  Go/No-Go  data  from 
criterion  referenced  tests  were  also  examined  but  found  to  be  essentially  all 
"go"  in  both  groups. 


TFTS  Effectiveness 


TFTS  students  received  course  grades  based  on  2  PE  and  2  written 
test  scores.  The  class  that  began  in  March  1983  was  told  that  their  PLATO 
test  scores  would  also  be  included  in  the  average.  As  a  result,  students 
treated  PLATO  exercises  even  more  seriously.  Interestingly,  their  PLATO 
scores  remained  the  same,  but  written  and  PE  scores  increased  considerably 
(see  below).  Since  this  change  in  procedures  (announcing  that  PLATO  scores 
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would  be  averaged  into  the  grades)  was  not  part  of  the  original  experimental 
design,  Battelle  captured  only  the  following  averaged  data  from  the  last 
three  classes  in  the  evaluation  period. 


Overall  PLATO-Only 

Class  Information  Grade  Grade 


Class  4 

Started 

N 

Avg. 

Range 

Avg. 

Range 

105 

Dec.  82 

9 

84.9 

79-88 

86.2 

83-89 

001 

Jan.  83 

7 

86.6 

82-94 

86.0 

S2-89 

002 

Mar.  83 

6 

94.3 

94-95 

86.2 

85-88 

The  sample  sizes  are  too  small  to  draw  firm  conclusions,  but 
anecdotal  information  from  the  instructors  indicates  a  significant  improve¬ 
ment  of  an  already  excellent  situation  after  the  announcement  was  made.  All 
of  the  above  scores  should  be  viewed  as  very  commendable:  the  criterion 
level  for  passing  the  course  is  70. 


Subjective  Effectiveness  Data 


Training  effectiveness  was  also  measured  via  ratings  by  students 
and  instructors.  TFTS  students  comparing  PLATO  to  the  real  equipment  felt 
they  learned  equally  from  each  and  felt  they  could  learn  "a  great  deal"  if 
they  had  learned  only  from  PLATO.  HAWK  students  felt  they  learned  less  from 
PLATO  and  could  have  learned  "a  little"  using  only  PLATO.  TFTS  instructors 
found  PLATO  greatly  increased  student  learning  and  greatly  increased  learning 
about  the  relationships  among  system  components  and  tests.  HAWK  instructors 
saw  little  or  no  change  in  either  of  these  categories. 

Speed  of  learning  for  TFTS  students  was  self-rated  as  faster  on 
PLATO,  but  HAWK  students  felt  they  learned  slower  on  PLATO. 

Simulation  fidelity  or  realism  is  an  important  dimension  of  training 
effectiveness  in  cases  where  a  training  simulation  is  used.  Most  (22)  TFTS 
students  stated  they  were  never  confused  by  differences  between  real  and  simu¬ 
lation  equipment  and  several  (12)  said  they  occassionally  were  confused.  None 
reported  being  confused  often.  These  students  were  divided  between  the  two 
highest  ratings  for  the  fidelity  and  adequacy  of  PLATO  for  training.  HAWK 
students  were  divided  fairly  evenly  between  those  who  were  never  confused  by 
differences  between  real  and  simulated  equipment  and  those  who  occassionally 
were.  Four  students  reported  being  confused  often.  Opinions  on  the  adequacy 
of  simulation  fidelity  for  training  spanned  the  range  of  alternatives.  Overall 
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the  ratings  were  slightly  positive. 

Students  answered  one  rating  question  about  the  relevancy  or 
applicability  of  PLATO  learning  to  the  actual  job.  TFTS  students  gave  PLATO 
very  high  marks  here,  HAWK  students  were  neutral  to  slightly  negative. 
Instructors  had  a  similar  question:  TFTS  instruction  got  positive  responses; 
HAWK  instruction  got  mostly  neutral  and  one  negative  response. 

Student  confidence  could  be  expected  to  increase  as  a  result  of 
using  the  simulators.  TFTS  responses  were  all  positive  or  neutral,  with 
half  the  students  reporting  PLATO  "increased  my  confidence".  HAWK  students 
were  almost  exactly  neutral.  A  similar  question  to  TFTS  instructors  produced 
all  positive  comments;  HAWK  instructors  comments  mirrored  in  reverse  the  TFTS 
results:  they  were  all  negative.  If  the  simulator  can  boost  a  student's 
confidence,  it's  also  possible  the  simulator  could  lead  to  overconfidence. 
However,  TFTS  instructors  found  student  overconfidence  decreased  (a  positive 
finding).  Three  of  the  four  HAWK  instructors  agreed,  but  one  reported 
increased  overconfidence. 

Because  students  don't  always  use  tech  ’"anuals  (TMs)  as 
frequently  as  they  ought,  one  goal  of  the  PLATO  simulations  was  to  increase 
this  use.  TFTS  instructors  reported  a  considerable  increase  in  TM  use  (both 
in  a  rating  item  and  in  open-end  comments),  but  HAWK  instructors  saw  no  change 
(3  of  4)  or  a  great  decrease  (1). 

One  of  the  most  consistent  findings  about  why  computer-based  train¬ 
ing  improves  performance  is  that  student  attention  is  enhanced.  Most  TFTS 
students  checked  the  highest  category  for  their  interest  level  while  using 
PLATO;  all  three  instructors  also  recorded  greatly  increased  attention. 

HAWK  students  reported  a  rather  high  interest  level,  only  slightly  lower  than 
TFTS  students.  HAWK  instructors,  however,  noted  no  change  or  even  a  decrease 
in  attention. 

The  student  questionnaire  contained  one  item  (number  3)  that  was  an 
overall  rating  of  effectiveness  and  acceptance.  On  a  1  to  10  scale  (10  being 
highest)  students  rated  the  usefulness  of  PLATO  simulation.  TFTS  student's 
ratings  were  all  positive  (i.e.,  the  lowest  rating  was  6),  the  mode  was  10,  and 
the  mean  was  8.9.  HAWK  student's  ratings  spanned  the  range  from  1  to  10,  the 
mode  was  10,  and  the  mean  was  only  5.7. 
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TABLE  0.  INDEX  OF  QUESTIONNAIRE  ITEMS  AND  SUBJECTIVE 
TRAINING  EFFECTIVENESS  VARIABLES 


Subjective  Training 


Effectiveness  Variables 
(Keyed  to  Tables  1  and  2a 
in  Appendix  A) 

Student 

End-Of-Course 

Evaluation 

Instructor 

End-Of-Course 

Evaluation 

1.2 

Perceived  amount  of  learning 

Questions  5  &  12 

Questions  6  &  17 

1.3 

Perceived  speed  of  learning 

Question  6 

-- 

1.4 

Perceived  fidelity  of 
simulation 

Questions  4  &  11 

-- 

1.5 

Perceived  application  of 
training  to  job  perform¬ 
ance 

Question  10 

Question  12 

1.6 

Perceived  extent  of  student 
confidence 

Question  8 

Question  9 

1.7 

Perceived  extent  of  student 
overconfidence 

-- 

Question  14 

1.8 

Perceived  use  of  TM‘s 

-- 

Question  10  ; 

1.9 

Perceived  holding  of  student 
attendion 

Question  9 

Question  8 

Training  Efficiency 


Training  efficiency  questions  as  defined  in  this  study  address  two 
issues.  The  first  is,  "Was  the  available  practical  exercises  (PE)  time  well 
used?"  Instructors  in  TFTS  found  a  lower  workload  and  more  time  for  helping 
students  when  they  had  PLATO  available,  but  HAWK  instructors  had  essentially 
an  opposite  view.  TFTS  students  felt  their  time  was  used  better  or  perhaps 
much  better  with  PLATO  whereas  HAWK  students  perceived  no  difference. 
Instructor  opinions  matched  those  of  their  students. 

The  second  efficiency  question  is,  "Was  the  simulation  itself  effi¬ 
cient?"  To  measure  this,  the  evaluation  plan  made  two  assumptions:  the 
simulated  tasks  should  not  take  longer  than  the  actual  tasks,  and  greater 
individualization  (because  of  feedback,  help,  etc.)  should  make  the  ratio 
between  fastest  and  slowest  student  larger  for  an  efficient  simulation  than 
for  the  real  task.  Only  TFTS  had  a  subject  for  which  the  above  comparisons 
could  be  reasonably  made,  and  only  two  instructors  had  enough  experience  to 
accurately  estimate  the  time  needed  to  perform  the  tasks.  The  first  hypothe¬ 
sis  was  upheld:  simulated  operation  was  as  fast  or  faster  than  actual  opera¬ 
tion.  However,  the  second  hypothesis,  that  PLATO  feedback  would  increase  the 
disparity  between  fast  and  slow  students,  was  not  observed.  In  fact,  the 
students  who  were  slowest  on  the  real  equipment  apparently  received  the 
greatest  "speed-up"  when  using  PLATO:  instructors  estimated  that  for  the 
slowest,  simulated  operation  took  1.0  hours  whereas  on  actual  equipment  the 
operation  took  1.5  hours. 


TABLE  1.  INDEX  OF  QUESTIONNAIRE  ITEMS  AND 

SUBJECTIVE  TRAINING  EFFICIENCY  VARIABLES 


Subjective 

Student 

Instructor 

Efficiency  Variables 

End-Of-Course 

End-Of-Course 

(Table  3,  Appendix  A) 

Evaluation 

Evaluation 

1.2 

Use  of  instructor's  time 
during  P.E. 

-- 

Questions  11  &  16 

1.3 

Use  of  student's  time  during 
P.E. 

Question  7 

Question  15 

1.4 

Estimated  time  to  complete 

-- 

Questions  25  -  28 

tasks 


Salvaged  Time 


Another  measure  of  efficiency  is  "salvaged  time>.  Since  typically 
only  half  the  students  could  work  with  the  real  TFTS  equipment  at  once,  the 
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time  of  the  other  students  was  being  underutilized  before  PLATO  simulations 
were  added  to  the  course.  Based  on  comments  from  course  personnel,  it  seems 
fair  to  count  as  "salvaged  time"  all  the  student  time  spent  using  the  TFTS 
simulations;  the  average  time  for'TFTS  students,  12.3  hours/person,  is  there¬ 
fore  regarded  as  the  amount  of  extra  instruction  each  student  received 
because  of  the  PLATO  simulations,  but  without  extending  the  length  of  the 
course.  TFTS  instructors  also  indicated  via  oral  comments  that  PLATO  totally 
eliminated  student  complaints  regarding  underuti 1 ized  time  that  occurred 
earlier.  HAWK  instructors  felt  because  real  equipment  was  available,  PLATO 
took  away  from  experience  using  it.  As  a  result  no  "salvaged  time"  would  be 
gained. 


Reduction  of  the  Need  for  Hardware 


One  possible  purpose  for  P LATO-type  simulations  is  the  replacement 
of  real  hardware  with  simulated  haroware.  To  measure  this  possibility  tne 
evaluation  plan  proposed  to  calculate  the  number  of  students  per  set  of  real 
hardware  before  and  after  PLATO.  In  actual  practice,  the  number  of  students 
was  more  a  function  of  class  frequency  and  class  size  than  of  hardware  avail¬ 
ability.  For  that  reason  and  because  there  is  no  value  against  which  to 
compare  the  ratio  obtained,  this  issue  was  examined  by  informal  interviews  of 
course  staff.  The  clear  consensus  in  the  TFTS  course  was  that  even  as 
valuable  as  the  PLATO  simulations  were,  some  minimum  time  with  the  real  equip 
ment  was  necessary  and  that  for  the  TFTS  course,  for  example,  a  further 
"dilution"  of  hardware  among  students  would  be  detrimental  to  the  training 
program.  They  pointed  out  that  it  was  not  infrequent  that  one  of  the  three 
TFTS  systems  was  broken.  Whereas  the  remaining  operable  equipment  could 
temporarily  accommodate  a  50%  increase  (from  one-third  of  the  load  to  one- 
half  of  the  load),  if  there  were  only  two  TFTS  units  when  one  failed,  tne 
remaining  unit  could  not  be  expected  to  handle  a  100%  load  increase  (from  one 
half  to  the  full  load). 


User  Acceptance 


User  acceptance  was  broken  into  two  parts;  student  acceptance  and 
instructor  acceptance.  The  responses  from  TFTS  and  HAWK  students  were  suffi¬ 
ciently  different  that  they  were  analyzed  separately. 


Analysis  of  TFTS  Student  Comments  from 
Open-End  Questionnaire  Items 


Two  questions  on  the  end-of -course  student  evaluation  asked  TFTS 
students  what  they  liked  and  what  they  disliked  about  the  PLATO  part  of  the 
course.  In  addition,  several  students  added  comments  of  a  similar  nature  to 
other  questions  and  to  the  end-of-week  evaluation  sheet.  All  these  comments 
were  divided  into  roughly  a  dozen  positive  and  a  dozen  negative  categories. 
The  total  number  of  comments  exceeds  the  number  of  students  because  most 
students  made  multiple  comments. 


Likes.  The  aspect  of  PLATO  that  most  student  comments  noted  (17) 
was  the  extra  simulated  practice  with  the  equipment.  A  large  number  (10)  also 
felt  that  use  of  the  computer  added  "interest"  and  "fun"  to  the  course.  Equal 
numbers  of  comments  (6)  were  received  for:  practice  with  the  TMs  using  a 
computer,  and  general  positive  (but  non-specific)  comments.  Five  comments 
showed  appreciation  of  the  graphics,  three  comments  each  dealt  with  the 
feedback  received  and  the  testing  aspects  of  the  computer.  Two  comments 
mentioned  interaction  and  one  each  noted  the  touch  input  and  the  reduced 
possibility  of  equipment  damage.  Thus,  in  general,  the  objectives  for  which 
PLATO  training  was  designed  were  noted  by  the  students.  The  comments  were 
really  quite  positive  (i.e.,  no  sarcastic  remarks)  and  no  unexpected  or 
unusual  comments  appeared. 

Pi  si i kes .  The  most  popular  negative  comment  was,  in  fact,  a  posi¬ 
tive  comment"!  F i fteen  comments  of  "none"  or  "nothing"  were  listed  as  dis¬ 
likes  about  the  PLATO  part  of  the  course.  The  second  most  frequent  comments 
regarding  dislikes  may  have  related  to  an  apparent  problem  with  differing 
tolerances  between  the  PLATO  and  the  TM  requirements.  Some  students  said 
merely  that  right  answers  were  sometimes  marked  wrong;  other  students  went  on 
to  explain  about  the  tolerances  problem.  Discussion  with  course  personnel 
indicated  that  the  tolerance  difference  was  a  deliberate  design  decision  to 
try  to  get  better  results  through  tighter  tolerances.  Four  people  commented 
about  touch  input  problems.  This  could  be  a  result  of  overly  restrictive 
judging  by  tne  software  (failing  to  account  for  parallax  and  sloppy  finger 
placement)  or  a  result  of  hardware  problems.  The  number  of  problems  noted  is 
predictable  for  touch  input.  Three  peoDle  made  what  is  probably  another 
example  of  positive  comment  in  a  negative  category:  they  felt  there  was  not 
enough  time  to  use  PLATO.  Three  comments  were  noted  regarding  difficulties 
in  moving  from  PLATO  simulator  tc  the  real  equipment.  Two  comments  each  were 
received  that  PLATO  feedback  was  confusing  and  that  the  computer  instruction 
was  too  simplistic.  Four  comments  were  made  only  once:  a  better  introduction 
to  using  the  computer  was  needed,  touch  input  combined  with  typing  was  confus¬ 
ing,  the  student  should  be  able  to  insert  errors  into  the  simulated  equipment, 
and  a  student  felt  as  if  he  were  just  playing  a  game  with  the  computer. 

TFTS  Summary.  Total  comments  about  likes  totaled  60;  those  about 
dislikes  totaled  42. Considering  that  all  of  the  likes  were  positive  com¬ 
ments  and  18  of  the  dislikes  were  positive  comments;  this  gives  a  ratio  of  78 
to  24  negative. 

Overall  these  comments  paint  a  picture  of  a  very  satisfied  student 
user  group.  It  might  be  noted  that  although  some  minor  courseware  rewrites 
might  be  needed  to  reduce  even  further  the  number  of  problems  noted  in  the 
comments,  no  problems  were  apparently  severe  enough  to  warrant  writing  them 
down  in  the  problem  log. 


Analysis  of  HAWK  Student  Comments 
from  Open-End  Questionnaire  Items 


The  HAWK  students  who  used  PLATO  answered  the  questions  about  likes 
and  dislikes  rather  differently  than  the  TFTS  students. 
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Likes .  The  most  common  reason  for  liking  PLATO  (9  responses)  was 
that  "it  broke  the  routine/was  fun/a  good  toy".  Six  people  found  it  'aster 
to  use  and  five  enjoyed  working  with  PLATO  as  a  computer.  A  few  (4)  felt  "it 
makes  you  think/it  helps  to  understand  tne  circuitry"  ana  two  liked  it 
because  it  was  easy  to  use.  One  comment  each  was  received  that  a  student 
liked  the  PLATO  simulation  because:  it  saved  the  training  equipment,  it  was 
good  and  more  of  the  subject  should  be  on  PLATO,  that  it  was  NOT  good  for 
teaching,  and  that  it  was  good  if  integrated  with  the  rest  of  the  course. 

Pi  si i kes .  Two  types  of  comments  tied  for  the  most  common  dislikes: 
six  students  each  noted  "not  enough  reaiism/no  feeling  of  learning"  and 
"doesn't  give  enougn  information/does  too  much  automatically".  Several  com¬ 
ments  (4)  were  made  about  the  brevity  of  the  PLATO  experience  being  too  snort 
to  be  useful.  Three  students  found  nothing  to  dislike  and  three  said  PLATO 
simulations  were  only  good  if  there  was  an  equipment  snortage  [tnere  isn't  a 
shortage  currently  in  tne  HAWK  course].  One  comment  each  was  received  tnat 
PLATO  was  "boring",  "not  detailed  enougn"  and  "mixed  up". 

HAWK  Summary.  Total  positive  comments  were  3C  and  negative  were  25. 
Moving  negative  comments  under  "likes'  to  "dislikes"  and  vice-versa,  one  finds 
there  were  32  positive  and  23  negative  comments.  Despite  their  numerical 
majority,  the  positive  comments  were  ofren  wea<  (e.g.,  the  most  popular  com¬ 
ment),  and  the  negative  comments  were  strong.  Cveral1  it's  clear  the  HAWK 
students  felt  PLATO'S  advantages  were  tncse  of  an  interesting  diversion  from 
learning;  few  comments  suggested  that  ii  had  an  important  role  in  training. 

The  HAWK  student's  exoerience  witn  PLATO  was  much  different  and  mucn  less  posi¬ 
tive  than  that  for  TFTS. 


Ranked  Acceptance  Items:  Students 


Two  end-of-course  items  rated  student  acceptance  on  a  5-point  scale. 
TFTS  students  voted  overwhelmingly  that  they  would  like  to  use  PLATO  for  otner 
hands-on  training;  most  also  said  zney  would  recommend  to  a  friend  that  a 
PLATO  course  be  selected  over  a  non-PLATO  course  if  convenient  and  many  said 
they  would  advise  the  friend  to  "fignt  tooth  and  nail  to  get  into  the  PLATO 
course".  These  are  Doth  very  strong  recommendations.  Most  HAWK  students,  in 
contrast,  felt  they  would  prefer  not  to  take  more  PLATO  in  other  training  or 
were  undecided  (there  was  not  a  strong  consensus).  Similarly,  they  were 
neutral  to  slightly  negative  about  advising  a  friend  with  respect  to  other 
PLATO-conta in ing  courses. 


TABLE  2.  INDEX  nr  QUESTIONNAIRE  ITEMS  AND 
STUDENT  ACCEPTANCE  VARIABLES 


Student 


Acceptance 

Variables 

vTable  4,  Appendix  A; 

Student 

End-Of-Course 

Evaluation 

1.1 

Likes  and  dislikes  about 
PLATO  training 

Questions  1  &  2 

1.2 

Willingness  to  use  PLATO 
in  other  P.E.  lessons 

Question  14 

1.3 

Willingness  to  take  other 
courses  involving  PLATO 

Questions  15 

Acceptance  Items:  TFTS  Instructors 


Instructor  acceptance  attitudes  matched  student  attitudes  fairly 
closely.  TFTS  instructors  each  listed  multiple  advantages:  student  self- 
confidence,  lack  of  equipment  damage,  saved  instructor  time,  individual  feed¬ 
back  to  students,  increased  attention,  etc.  Two  instructors  found  no  dis¬ 
advantages  and  a  third  noted  tnat  use  of  PLATO  meant  less  time  on  actual 
equipment  (but  qualified  his  comment  saying  the  advantages  outweighed  the 
disadvantages).  An  open-end  question  about  role  changes  caused  by  a  wider 
PLATO  implementation  produced  two  similar  positive  comments  of  "more  time  to 
give  individual  attention"  and  one  comment  predicting  little  role  change.  A 
rating  question  measuring  existing  role  changes  was  very  positive.  An  open- 
end  question  about  training  practices  elicited  explanations  of  implementation 
procedures  and  reiterations  of  advantages. 

TFTS  instructors  had  consensus  at  the  highest  positive  level  regard¬ 
ing  more  PLATO  simulations  for  their  course  and  for  other  courses  at  the 
school.  Regarding  their  seeking  a  position  as  instructor  in  a  PLATO-based 
course,  their  responses  were  strung  evenly  between  a  neutral  and  very  positive. 


Acceptance  Items:  HAWK  Instructors 


HAWK  instructors  responding  to  advantage/disadvantage  questions 
found  no  PLATO  advantages  (3)  and  use  possibly  for  remediation  (1). 
Disadvantages  were  plentiful:  six  disadvantages  were  cited  by  four  instruc¬ 
tors.  They  all  dealt  with  the  extra  time  and  manpower  needed  to  implement 
PLATO.  Considering  the  amount  of  terminal  familiarization  time  compared  to 
the  student  learning  time,  the  instructor's  comments  are  certainly  under¬ 
standable.  An  open-end  question  about  role  changes  caused  by  a  wide  imple¬ 
mentation  of  PLATO  in  the  course  resulted  in  responses  nearly  identical  in 


20 


substance  to  the  "disadvantages"  question.  A  rating  question  mt.'^ur’ng  exist¬ 
ing  role  change  was  neutral  to  negative. 

Instructor's  comments* to  an  open-end  question  about  changes  in 
training  practices  resulting  from  PLATO  had  only  negative  comments:  time 
lost  from  real  equipment,  students  lax  in  procedures  after  using  PLATO  (2), 
and  students  less  safety  conscious  (touching  on  PLATO  was  theorized  to  induce 
students  to  touch  actual  components). 

,-iAWK  instructors  were  mostly  undecided  aDout  advocating  PLATO 
simulation  for  otner  courses  at  tne  scnool ,  but  definitely  did  not  want  tnem 
in  the  HAWK  course.  Of  the  four  instructors,  two  would  avoid  if  possible 
teaching  a  course  using  PLATO  simulation  3no  two  would  neither  request  nor 
avoid  such  a  course. 


TABLE  3.  INDEX  OF  QUESTIONNAIRE  ITEMS  AND 
INSTRUCTOR  ACCEPTANCE  VARIABLES 


Instructor  ana 

School  Staff 

Acceptance  Variables 
(Table  5,  Appendix  A) 

Instructor 'Staff 

Ena-Qf-Course 

Evaluation 

1.1 

Advantages  and  disadvantages 
with  using  PLATO 

Questions  2  i  3 

1.2 

Willingness  to  expand  PLATO 
lessons  in  current  course 

Question  18 

1.3 

Willingness  to  teach  other 
courses  using  PLATO 

Questions  19  &  23 

1.4 

Perceived  changes  in  role  of 
i nstructcr/staff 

Questions  5  &  7 

1 . 5 

Perceived  changes  in  conduct 
of  training 

Question  4 

Quality  of  Implementation 


An  adequate  summative  evaluation  requires  that  some  attention  oe 
given  to  the  quality  of  implementation  of  the  courseware.  Unless  the 
implementation  is  carried  out  in  accordance  with  the  project  plans,  and  is  of 
high  quality,  conclusions  cannot  be  made  about  the  impact  of  the  courseware. 


Quality  of  implementation  was  assessed 
that  recorded  the  frequency  of  typical  problems 
incompletely-tested  courseware.  All  these  items 


by  means  of  seven  questions 

experienced  with 

were  listed  on  what  was 
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labeled  the  "end-of-week  evaluation".  In  both  courses  a  sinqle  end-of- 
week  evaluation  was  used,  even  if  the  PLATO  simulations  were  used  over  several 
weeks.  Also,  two  sliqhtly  different  *orms  of  th's  evaluation  were  used.  One 
course  used  an  earlier  draft  form  and  the  other  used  a  finalized  version. 

TFTS  students  had  three  response  options  for  describing  the  frequency  of 
occurrence  of  eacn  problem:  never,  once,  more  than  once.  HAWK  students  had 
the  same  questions,  but  were  just  given  a  yes/no  item,  "did  this  problem 
ever  occur",  followed  by  "i'"  yes,  how  often:  once,  two  or  three  times,  four 
or  more  times1'. 

A  subjective  reading  of  the  TFTS  results,  based  on  Battelle's 
experience  and  based  upon  analysis  of  student's  open-end  comments  as 
explanations  of  the  above  implementation  data,  suggests  that  most  of  the 
"bugs"  had  been  removed  prior  to  student  testing  and  that  efforts  toward 
further  polishing  of  the  courseware  would  probably  not  nave  influenced  the 
results.  The  only  cases  in  which  a  majority  of  students  had  a  problem  more 
than  once  seemed  to  be  caused  by  non-responsi ve  touch  panels  and  needing  an 
instructor  for  further  information.  Considering  the  number  of  hours  on  PLATO, 
the  predictability  of  responses  (question  4),  the  indexing  of  content 
(question  6),  and  the  simulation  fidelity  (question  8)  all  received  very  high 
tallies  in  the  "never  a  problem"  category. 

HAWK  student  data  are  difficult  to  compare  directly  to  TFTS  data 
since  the  duration  of  the  PLATO  experience  was  so  different.  However,  the 
pattern  of  responses  for  HAWK  is  basically  similar,  with  fewer  problems 
noted  because  of  fewer  hours  of  operation.  Two  exceptions  to  the  pattern 
were  that  HAWK  students  recorded  more  touch  panel  problems  (possibly  a  hard¬ 
ware  or  programming  problem)  and  simulation  fidelity  problems.  The  higher 
rate  of  problems  recorded  for  HAWK  could  have  contributed  to  the  lower 
acceptance  of  PLATO  by  students  in  that  course,  particularly  since,  for 
example,  10  of  the  34  students  reported  problems  with  simulation  fidelity 
that  occurred  two  or  more  times  during  their  short  usage.  An  alternate 
explanation  is  that  the  high  number  of  problems  noted  was  a  symptom  of 
dissatisfaction  arising  from  some  other  cause  rather  than  a  contributory 
factor  to  the  cause. 

An  instructor  questionnaire  item  (number  24)  measuring  the  quality 
of  implementation  in  terms  of  instructor  assistance  required  produced  posi¬ 
tive  results  for  TFTS  and  negative  results  (more  assistance  needed)  for  HAWK. 


General izabil ity 


There  are  two  questions  that  might  be  termed  "general izabil ity" ; 
first,  "Are  the  results  valid,  and  thus  general izable  to  other  students  and 
courses?"  and  second,  "Based  on  the  knowledge  and  experience  of  the  instruc¬ 
tors  and  school  staff,  are  there  other  training  situations  that  would  benefit 
from  application  of  this  technology?"  Although  both  questions  are  related, 
the  first  question  is  not  a  measurable  quantity,  but  is  based  on  the  quality 
of  the  imolementation  and  the  quality  of  the  research  design,  whereas  the 
second  question  was  hoped  to  be  measurable  to  some  extent  using  subjective 
ratings. 
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Two  rating  items  attempted  to  easure  tne  ;nstructor-s  impressions 
of  the  general izabil ity  of  the  PLATO  simulation  approach  within  tneir  own 
course  (question  21)  and  to  other  courses  (question  22).  TF7S  staff  found 
PLATO  applicable  to  all  other  P.E.  lessons  in  7F7S  and  "almost  all"  similar 
situations  in  other  courses  with  limited  equipment  for  training.  (Note:  the 
HAWK  course  does  not  suffer  from  equipment  shortages  as  does  the  TFTS  course.) 

The  HAWK  instructors  (3  of  4)  said  PL4T0  was  not  applicable  to  other  PE  les¬ 
sons  and  some  (2  of  4)  said  PLATO  shouldn't  have  been  used  where  it  was. 

Their  opinion  about  applicability  to  other  courses  was  also  negative:  3  of  4 
found  only  a  limited  number  of  courses  that  might  benefit  and  one  instructor 
found  no  use  at  all  for  PlATO  simulation.  Clearly  instructor  experience 
strongly  colors  opinion,  even  about  applications  outside  their  own  course. 

The  total  lack  of  instructor  consensus  demonstrates  that  inadequate  data  are 
available  with  wnich  to  form  a  conclusion  on  general  izabil  ity ,  and  tr.at  instructor 
data  alone  are  probably  not  adequate  to  settle  this  issue. 


Rel iaoi 1 i tv 


Failure  Rates 


Because  of  the  excellent  performance  of  tne  hardware  and  software, 
many  of  the  reliability  measurements  nave  little  meaning  or  cannot  be  calcu¬ 
lated.  As  stated  in  the  test  plan,  tne  software  problems  encountered  during 
the  very  first  class  should  not  be  iicluded  in  the  calculation  of  mean  time 
between  problems  (MTBP).  Since  no  problems  were  logged  after  the  first  week, 
this  value  cannot  oe  calculated.  Tne  only  software  errors  Togged  were  four 
noted  during  the  first  TFTS  class. 

As  a  result  of  the  above,  the  three  metrics:  mean  time  between 
failures  (MTBF),  mean  time  between  problems  (MTBP)  and  mean  time  to  repair 
(MTTR)  as  well  as  the  malfunction  profile  are  not  calculable  or  reported  here. 
The  only  terminal  hardware  failure  was  a  toucn  panel  that  took  exactly  one 
week  to  repair. 

Perceived  Rel iabi 1 i ty 


In  addition  to  various  measures  of  reliability  it  is  also  instruc¬ 
tive  and  important  to  measure  perceived  reliability  since  a  few  failures  at 
critical  moments  may  disproportionately  affect  user's  perceptions  of  reli¬ 
ability.  TFTS  students  rated  PLATO  reliability  (question  13)  as  much  better 
than  other  training  equipment.  HAWK  student's  ratings  were  positive  to  neu¬ 
tral,  although  no  reliability  problems  were  logged  in  that  course.  The 
three  TFTS  instructors  (question  20)  saw  PLATO  reliability  as  much  better 
than  other  training  equipment;  HAWK  instructors  saw  it  as  "much  better"  or 
about  the  same  (1). 
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PLATO  System  Availability 


Availability  is  the  percentage  of  time  the  system  is  available  to 
deliver  training.  Availability  will  be  reported  separately  for  each  type  of 
terminal,  but  the  values  will  mean  slightly  different  things  in  an  operational 
sense.  If  one  had  90S  availability  on  10  micro-PLATO  systems,  one  could 
expect  on  the  average  that  one  system  would  be  inoperable.  About  IS  of  the 
time,  two  systems  might  be  inoperable.  Very,  very  rarely  would  they  all  be 
inoperable  at  once.  If  one  had  central  PLATO,  all  failures  were  central 
system  problems  (no  inoDerable  terminals)  and  90S  reliability,  one  would  have 
no  terminals  operable  IQS  of  the  time.  Hence  90S  availability  may  describe 
both  systems  accurately,  yet  have  very  different  implications  operationally. 

The  HAWK  terminal's  usaqe  was  combined  with  TFTS  usage  for  all 
analyses.  Availability  was  calculated  according  to  the  following  formula: 


Avail . 


^Failures 

t 

#  Hrs.  down  (8  hrs./day)  x  ^Terminals  affected) 


1 


^Working  days 

8  x  ^Terminals  available 

1 


x  100% 


For  micro-PLATO  system,  central  PLATO  system  downtime  is  ignored. 
For  that  system  of  five  terminals,  availability  was  calculated  to  be  99.17T. 
For  central  PLATO  based  systems,  both  local  hardware  problems  and  central 
PLATO  system  problems  are  included.  For  a  system  of  seven  terminals,  two  of 
which  were  connected  to  central  PLATO,  availability  was  99.20%. 


Amount  of  Use 


The  amount  the  terminals  were  used  may  have  had  a  bearing  on  how 
much  maintenance  was  required  and  how  frequently  intermittent  problems 
(including  central  system  downtime)  were  noticed.  Two  kinds  of  usage  were 
discernible  from  logs  of  usage:  student  use  and  author  use.  For  HAWK  there 
was  no  usage  of  the  terminals  by  authors.  Student  usage  from  11  May  through 
18  October  on  one  terminal  totaled  20.25  hours. 


Usage 


Usage  as  defined  by: 
#Uses 


^ 7  Duration  in  hours 

_ 

*Weeks  x  ^Terminals  available 


(In  units  of  hours/terminal/week) 
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for  HAWK  was  0.84  hours/week/terminal.  For  TFTS,  two  terminals  were  con¬ 
nected  to  central  PLATO  for  authoring  purposes  and  were  not  to  be  used  by 
students  unless  necessary  because  of  equipment  failure.  No  such  failure 
occurred  and  logged  usage  indicated: 

Terminal  *  #Hours  Student  Usage  -fHours  Author/Demo  Use 

1 

80:40 

0 

7 

39:10 

274:00 

3 

0 

900:55 

4 

0 

23:20 

5 

32:35 

0 

6 

87:45 

1 :50 

Total 

240:10 

1200:05 

The  four  TFTS 

student  terminals  compiled 

usage  of  10  hours/week  or 

2.50  hours/week/terminal 

The  four  TFTS  terminals 

usee  for  authoring  or 

demonstrations  had  an  average  usage  of  12.5  hours/week/terminal ,  with  termina 
43  in  "fulltime"  use  37.5  hours  per  week. 

With  a  higher 

percentage  of  terminal  time 

■  in  use,  it  mignt  be 

decrease. 


A  further  analysis  of  tne  usage  logged  revealed  tne  following:  The 
log  was  apparently  maintained  very  carefully.  Very  few  data  were  lost  due  to 
incomplete  records,  etc.  Session  length  varied  Py  usage  type.  HAWK  students 
use  varied  from  1  hour  to  2-3/4  hours  with  the  two  modes  at  1  nour  to  2-3/4 
hours.  TFTS  students  typically  naa  brief  sessions:  5-15  minutes  was  not 
uncommonly  short  and  20-40  minutes  typical.  Only  a  few  students  spent  long 
blocks  of  time  (2-6  nours)  at  tne  terminal.  Author  usage  tended  to  be  at 
least  an  hour  in  length,  but  6-8  hours  was  not  uncommon.  Off-hours  use  was 
most  typically  author  usage.  Since  so  few  failures  were  recorded,  off-hours 
use  was  not  separately  analyzed. 


Cost  Extrapolation  and  Analysis 


For  TFTS  and  HAWK,  the  costs  were  figured  similarly.  Most  cost 
items  were  based  on  quotes  from  current  GSA  rates  combined  witn  estimates 
about  the  time  and  resources  needed  for  software  maintenance  to  keep  train¬ 
ing  materials  up-to-date.  For  each  course,  the  minimum  configuration  pos¬ 
sible  was  costed.  Thus,  such  items  as  on-line  PLATO  services  (connection 
to  a  large  time-sharing  system),  printers,  etc.,  were  eliminated.  Originally 
for  TFTS,  it  was  estimated  that  six  (6)  stand-alone  micro-PLATO  terminals 
would  be  needed  for  a  class  of  12  students.  During  implementation  it  was 
found  chat  four  (4)  terminals  were  sufficient.  The  HAWK  course  has  used  a 
single  terminal  throughout  the  test  period.  Costs  were  calculated  based  on 
1982  constant  dollars;  that  is,  1982  costs  were  extended  for  the  estimated 
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10  years  of  life  of  hardware  and  software  with  the  assumption  that  mainten¬ 
ance  costs  would  rise  at  the  same  rate  as  inflation.  Thus  no  inflation  rate 
nor  time-value  of  dollars  have  been  explicitly  included  in  the  calculation. 

The  one  cost  that  could  not  be  determined  exactly  is  the  1982  cost  of  a  micro- 
PLATO  terminal  on  a  GSA  schedule.  The  manufacturer  has  a  different  model  than 
used  in  this  test  (equivalent,  but  newer)  on  the  GSA  schedule  at  $6,780.  The 
manufacturer  has  the  exact  model  used  in  this  test  available,  only  at  retail, 
for  $7,450.  In  our  calculations ,  the  $6,780  figure  was  used.  For  TFTS  the 


calculation  for  minimum  cost  is: 

First  Year 

Four  (4)  Terminals  x  $6,780  $  27,120 

Maintenance  for  four  (4)  Terminals  at  $81/Terminal/ 

Month  (3  months  free  maintenance  with  purchase) 

4  x  9  x  $81  2,916 

20  Hours  Software  Maintenance  x  $31. 50/hr.  630 

Disk  space  rental  $4/Month/Space  x  12  x  70  spaces  3,360 

Remastering  Disks  and  Copying  _ 1_36 

Total  $  34,162 


Second  through  Tenth  Year 


Maintenance  for  four  (4)  Terminals  at  $81/Terminal/ 
Month 

20  Hours  Software  Maintenance  x  $31. 50/hr. 

Disk  Space  Rental  $4/Month/$pace  x  12  x  70 
Remastering  Disks  and  Copying 
Total 

x  Nine  (9)  Years 
10  Year  Grand  Total 


$  3,888 

630 
3,360 
136 
$  8,014 

72,124 
$106,288 


The  costs  for  a  terminal  for  the  HAWK  course  are  one-fourth  those 
for  TFTS,  or  $26,572.  The  incremental  cost  for  additional  terminals  (should, 
for  example,  the  student  flow  rate  in  the  TFTS  course  increase  so  much  that 
more  computer  terminals  were  needed)  is  $16,597  per  terminal  over  the  10 
years  for  either  HAWK  or  TFTS. 

Developmental  costs  were  not  addressed  by  the  test  and  evaluation 
plan,  but  some  data  gathered  as  part  of  the  evaluation  paint  a  broad  picture 
of  these  costs.  The  vendor,  CDC,  was  paid  $93,845  to  develop  the  purchase 
of  HAWK  and  TFTS  materials.  Not  included  in  that  amount  were  hardware  and 
computer  time  used  by  CDC  and  MMCS  staff  in  development.  No  estimates  of  the 
staff  time  or  other  resources  used  at  the  MMCS  were  made. 

Important  cost  questions  which  cannot  be  explicitly  answered  are, 
"Can  a  PLATO  microcomputer  that  costs  under  $10,000  each  replace,  for  training 
purposes,  TFTS  components  which  cost  over  $100,000  each,"  and,  "Can  use  of 


PLATO  simulation  reduce  equipment  damage?"  Although  these  questions  cannot 
be  answered,  information  addressing  these  issues  was  gathered  as  part  of  this 

evaluation.  For  training  a  class  of  12  students  JF^0un155Q!r§05eeded‘ 

Purchase  price  for  these  three  units  is  estimated  at  $345,000  -  $400,000. 
Procurement  of  these  units  is  slow  and  difficult  compared  to  procurement  of 
PLATO  microcomputers.  Because  spare  parts  for  the  TFTS  units  are  difficu  t  to 
obtain,  downtimes  tend  to  be  long  (especially  compared  to  that  of  PLATO  si mu 
latorsj.  During  the  time  Battelle  visited  MMCS  to  collect  data,  one  TFTS  unit 

was  down. 

One  of  the  instructor  questions  asked  if  damage  to  equipment  changed 
as  a  result  to  PLATO  simulation  usage.  The  analysis  showed  that  the  two  TFTS 
instructors  who  responded  both  felt  that  PLATO  simulations  greatly  decreased 
damage  whereas  3  of  4  HAWK  instructors  noted  little  or  no  difference  and  one 
felt  damage  had  increased. 

Interviews  revealed  no  indications  that  significant  changes  had 
occurred  to  the  number  of  safety  violations  nor  was  any  equipment  dPP^^'y 
damaged  via  student  error.  Lack  of  equipment  damage  is  a  finding, 

but  not  conclusive  since  there  was  no  control  group  and  equipment  damage  is 
not  a  frequent  event. 
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CONCLUSIONS  AND  FINDINGS 


The  conclusions  and  findings  differ  sharply  between  the  two  courses 
in  which  PLATO  simulations  were  implemented.  Considering  that  the  training 
materials  for  both  courses  were  developed  by  the  same  team  of  people,  the 
difference  is  even  more  dramatic.  It  is  dangerous  to  make  statements  that 
are  broad,  based  on  data  collected  from  so  few  instructors  (3  TFTS,  4  HAWK), 
but  a  few  observations  seem  warranted  based  on  the  high  degree  of  consensus 
found  within  each  of  the  two  sets  of  instructor  data.  Without  exception,  the 
instructor  and  student  data  within  a  course  supported  each  other.  With  few 
exceptions,  the  data  from  one  course  contradicted  the  data  from  the  other 
course.  In  general,  the  instructor  data  (admittedly  few  in  number)  tended  to 
represent  a  more  extreme  view  than  did  student  data.  It  would  be  too  strong 
to  say  that  this  showed  students  were  mirroring  instructor  attitudes;  on  the 
other  hand,  some  research  studies  have  indicated  that  students  are  strongly 
cued  by  instructor  attitudes. 

The  basis  for  the  instructor  and  student  attitudes,  whatever  their 
source,  seems  rooted  in  two  factors:  (a)  the  original  purpose  for  introducing 
PLATO  simulators  and  (b)  size  of  the  implementation.  In  the  TFTS,  there  was 
a  considerable  need  for  some  technique  or  technology  to  solve  the  problem  of  a 
shortfall  of  equipment.  Other  alternatives  than  PLATO  nay  have  been  as  welcome 
and  effective  if  they  addressed  this  problem.  For  the  TFTS  course  a  consider¬ 
able  number  of  hours  of  PLATO  instruction  are  available  (about  12).  Therefore 
the  time  "invested"  to  figure  out  how  to  run  this  new  device  could  be 
"amortized"  over  a  considerable  "payback"  period. 

For  the  HAWK  course  the  above  two  factors  were  reversed.  The  HAWK 
course  was  working  fine  as  it  stood.  Plenty  of  equipment  was  available. 

Instead  of  implementing  PLATO  as  part  of  a  new  course  (as  in  TFTS),  implement¬ 
ing  PLATO  in  the  HAWK  course  meant  discarding  some  existing  training  so  that 
PLATO  could  be  substituted.  The  best  PLATO  could  do  in  the  HAWK  course  was  a 
little  better  since  the  current  instruction  was  at  least  adequate.  If  PLATO 
did  nothing  at  all  or  functioned  in  the  mediocre  way  tnat  the  first  draft  of 
instruction  materials  often  do,  it  would  be  worse  than  the  previous  situation. 

In  TFTS,  the  best  PLATO  could  do  would  be  to  "save  the  day"  with  respect  to 
delivering  the  course  while  suffering  an  equipment  shortage,  if  it  did  nothing, 
it  wouldn't  be  much  more  than  a  blind  alley  to  be  abandoned  in  a  searcn  for 
an  eauipment  shortage  solution. 

With  those  constraints  understood,  the  TFTS  results  are  that  the 
PLATO  simulations  were  effective,  efficient,  accepted  by  students  and 
instructors,  reliable,  and  potentially  cost-effective.  The  HAWK  results  are 
that  the  PLATO  simulations  are  equally  effective  (test  results)  or  less 
effective  (subjective  results),  of  questionable  efficiency,  unacceptable  to 
students  and  instructors,  reliable,  and  less  likely  cost-effective.  The 
lower  potential  for  cost-effectiveness  is  because  the  large  number  of 
terminals  needed  to  quickly  serve  a  class  would  sit  unused  for  long  periods. 
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APPENDIX  A 

DESCRIPTION  OF  EVALUATION  VARIABLES 
TABLES  1  THROUGH  9 


TABLE  1.  TRAINING  EFFECTIVENESS  VARIABLES  -  TOW  COURSE 
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APPENDIX  B 

TFTS  STUDENT  EVALUATION  FORMS  AND  RAW  DATA 
END-OF-WEEK  EVALUATION 
END-OF-COURSE  EVALUATION 


1 


STUDENT 

END-OF-WEEK  EVALUATION 


You  ha ve  just  completed  course  modules  using  PLATO, 
as  best  you  car. ,  so  that  ve  can  improve  the  modules, 
only  apply  to  this  week’s  study. 

Course  Name:  TFTS 


Pusase  gt~i.  tn  tats  form. 
Your  answers  should 


Today's  Date: 


month 


1.  Which  week  of  this  course  have  you  just  completed? 


Pi-ease  answer  tr.e  cuesttcns  oetow. 


~ae  croc  terns,  snow 


how  often .the  problem  came  up  in  your  use  of  PLATO  during  the  week,  remember, 
your  answers  should  only  be  for  the  week  you  have  just  completed. 

2.  Were  you  ever  unsure  of  what  to  do  next  on  PLATO  (for  example,  touch  the 
screen,  hit  a  key,  or  wait  for  PlATO)?  No  14  Yes  '  ■ 

If  yes,  how  often  did  this  happen? 

Once _ 5_  Two  or  three  times  3  Four  or  more  times  4 

3.  Were  you  ever  sure  you  gave  the  right  answer,  but  PLATO  said  you  were 

wrong?  No  Yes  10 

If  yes,  how  often  did  this  happen? 

Once _ 3_  Two  or  three  times  3  Four  or  more  times  - 

4.  Did  PLATO  ever  do  something  that  surprised  you  (for  example,  you  hit 
BACK,  but  PLATO  went  forward)?  No  2i  Yes  7 

If  yes,  how  often  did  this  happen? 

Once _ [_  Two  or  three  times  *  Four  or  more  times  ' 

5.  Did  you  ever  hit  a  key  or  touch  the  panel,  and  nothing  happened? 

No  IS  Yes  17 

If  yes,  how  often-  did  this  happen? 

Once  3  Two  or  three  times  >2  Four  or  more  times 
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Was  it  ever  hard  to  study  or  practice  a  test  on  PLATO  because  you 
couldn't  get  to  it,  or  because  you  had  to  go  through  too  many  other 
steps  to  get  to  the  part  you  wanted?  No  30  Yes  £ 

If  yes,  how  often  did  this  happen? 

Once  l  Two  or  three  times _  Four  or  more  times 


Did  you  ever  have  to  ask  your  instructor  for  help  because  PLATO  was  not 
correct  or  was  incomplete?  No  22  Yes  1 3 

If  yes,  how  often  did  this  happen? 


Two  or  three  times  6 


Four  or  more  times 


8.  Were  you  ever  confused  because  the  drawings  on  PLATO  looked  or  worked 
different  from  the  real  equipment?  No  Yes 


If  yes,  how  often  did  this  happen? 
Once  2  Two  or  three  times  6 


Four  or  more  times  4 


STUDENT 

END-OF-COURSE  EVALUATION 


"Please  complete  this  form,  vo  the  best  of  your  ability  so  that  this  course  car 
be  fully  and  ~airlu  evaluated. 

r  M  TFTS 

Course  Name:  _ 

Today's  Date:  _  _  _ 

day  month  year 

1.  What  did  you  like  about  the  PLATO  part  of  the  course?  _ 


2.  What  did  you  dislike  about  the  PLATO  part  of  the  course? 


3.  On  a  scale  of  1  to  10  (with  10  being  the  highest  score),  how  useful  do 
you  think  the  PLATO  simulations  were? 

Circle  a  number:  1  2  3  4  5  6^  7'  3 ‘ ‘  9  10  ^ 

4.  Did  the  training  on  PLATO  ever  confuse  you  because  the  real  equipment 
looked  or  worked  differently  than  the  PLATO  simulations? 

Zl  1)  No,  it  never  did 

12  2)  Yes,  occassionally 

_  3)  Yes,  often 
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8.  When  you  used  PLATO,  how  mucn  do  you  tnink  you  learned  compared  with 
using  the  -eal  ecuipment? 

0  1)  Learned  much  more  using  ^LATC- 

s  2)  Learned  more  using  5LA~C 

3)  Learned  about  the  same  amount  using  PLATO 

"  A)  Learned  1  ess  using  PLATO 

•  5)  Learned  much  less  us"ing  PLAT0 

6.  When  you  used  pLAT0,  now  fast  do  you  think  you  learned  compared  with 
using  tne  real  equipment? 

_  j  1 }  Learned  much  faster  using  PLATO 

'  ~  2)  Learned  faster  using  PLATO 

~  3)  Learned  in  about  the  same  time  using  PLATO 

A  4)  Learned  slower  using  PLATO 

_  5)  Learned  much  slower  using  PLATO 

7.  When  vou  used  PLATO,  how  did  it  affect  the  use  of  your  time  during 
P .  E . '  s  ? 

_ 1 )  Made  much  better  use  of  my  time 

;•?  2)  Made  better  use  of  my  time 

_ £_  3)  Made  1 ittle  or  no  difference  in  use  of  my  time 

_ 4)  Made  worse  use  of  my  time 

_  5)  Made  mucn  worse  use  of  my  time 

8.  How  did  use  of  PLATO  influence  your  confidence  about  operating  the  real 
equipment? 

_ y_  1 )  Greatly  increased  my  confidence 

_ :j_  2)  Increased  my  confidence 

_ j_  3}  Did  not  change  my  confidence 

_  4)  Decreased  my  confidence 

_  5)  Greatly  decreased  my  confidence 

9.  How  interested  were  you  in  going  through  the  PLATO  lessons? 

_ J9_  1 )  Very  interested  while  using  PLATO 

_ o  2)  Interested  while  using  PLATO 

_  3)  Not  interested  while  using  PLATO 

_ Z_  4)  Bored  while  using  PLATO 

_  5)  Very  bored  while  using  PLATO 
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10.  How  much  of  what  you  learned  using  PLATO  could  oe  applied  or  tne  job? 

'  *  1 )  Almost  al 1  of  what  I  learned 
1 2  2)  Most  of  what  I  learned 

_ 5_  3)  Some  of  what  I  learned 

_ 4)  Little  of  what  I  lea  -neo 

_  5)  None  of  what  I  learned 

11.  For  training  purposes now  adequately  die  tne  PLATO  simulations  work  like 
tne  real  equipment? 

Id  1 )  Very  Adequate  for  training 
Id  ?}  Reasonably  adequate  for  training 

_ 5_  3j  Boraerl  : ne  for  training 

_ l_  4)  Somewhat  inadequate  for  training 

_  5}  Very  inadequate  for  training 

12.  How  much  do  you  tninK  you  could  have  learned  about  the  operation  of  the 
real  equipment  using  only  PlATG? 

_ j_  1)  Almost  everything 

1  a  2)  A  great  deal 

_ j_  3)  Some 

_ l_  4)  A  little 

_  5}  Almost  nothing  at  all 

13.  Compared  with  other  training  equipment  you  nave  used  (slide  projector, 
video  cassettes,  Beseler  Cue/See),  PLATO'S  reliability  was: 

12  1)  Mucn  better  tnan  otner  training  equipment 

_ l_j_  2)  Better  than  other  training  equipment 

_  3}  About  the  same  as  other  training  equipment 

_  4)  Worse  than  other  training  equipment 

_  5)  Much  worse  than  other  training  equipment 

14.  Would  you  line  to  use  PLATO  for  otner  hands-on  training? 

_ 2J_  1 )  Would  very  much  1 i ke  to  use  PLATO 

_ j  2)  Would  1  ike  to  use  PLATO 

_ 5  3)  Undecided 

_ l  4)  Would  not  1 i ke  to  use  PLATO 

_  5)  Would  very  much  not  1 i ke  to  use  PLATO 
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15.  How  would  you  advise  a  friend  who  had  a  choice  between  taking  a  course 
where  there  was  a  great  deal  of  PLATO  simulations  or  another  course 
that  covered  the  same  material  but  without  PLATO? 

?  i  1)  Fight  tooth  and  nail  to  get  into  the  PLATO  course 

; ;  2)  Request  PLATO  course,  if  convenient 

_ 3)  Neither  reauest  nor  cvoid  course 

_ j_  4)  Avoid  PLATO  course,  if  possible 

_  5)  Avoid  PLATO  course  like  the  plague 

16.  How  much  time  did  you  spend  on  PLATO  compared  with  other  students  in  your 
class? 

_ 4_  1 )  More  time  on  PLATO  than  others 

2i  2)  About  the  same  amount  of  time  on  PLATO  as  others 
_ 2  3)  Less  time  on  PLATO  than  others 


r* 


APPENDIX  C 

TFTS  INSTRUCTOR  EVALUATION  FORMS  AND  RAW  DATA 


PLATO  INSTRUCTOR/STAFF  QUESTIONNAIRE 

Please  complete  this  form  to  the  best  of  your  ability  so  that  this  course  can 
be  fully  and  fairly  evaluated. 

Course  Name:  TF TS _ 

Today's  Date:  _  _  _ 

day  month  year 


1.  Please  check  the  item  which  best  describes  your  role  in  this  course. 

3  1)  Regular  instructor  (Answer  all  questions.) 

_ 2)  School  staff  (Stop  after  Question  5.) 

Specify  position: _ _ 

2.  What  do  you  feel  are  the  instructional  advantages  of  using  PLATO  in  this 

particular  course?  _ _ _ _ 


3.  What  do  you  feel  are  the  instructional  disadvantages  of  using  PLATO  in 
this  particular  course? _ _ 


4.  What  changes  in  training  practices  occurred  in  this  particular  course 
because  of  the  use  of  PLATO?  _  _ 


5.  If  the  use  of  PLATO  became  more  widespread  (more  lessons  in  this 
course  and/or  more  courses),  how  would  your  role  be  changed?  _ 


In  Questions  6  through  2 servers  oraciical  exercises  u  sire  FLA  TO  wit': 
practical  exercises  using  onlu  real  equipment .  For  each  question,  please 
check  the  one  box  which  best  describes  your  opinion  about  the  impact  of  FLAT ' 
simulations  tn  this  course. 


6. 

7. 

8. 
9. 

10. 

11. 

12. 

13. 

14. 


How  did  use  of  PLATO  in  practical 
exercises  affect: 


Student  learning  during  P.E . 

Instructor's  confidence  and  feeling  of  use¬ 
fulness  during  P.E . 

Student  attention  during  P.E . 

Student  confidence  when  dealing  with  real 
equipment.  . 

Use  of  TM's . 

Instructor's  workload  during  P.E . 

Application  of  learning  to  job  performance.  ... 

Damage  to  equipment  items  used  for  training.... 

Student  overconfidence  when  dealing  with 
real  equipment . 


15.  Efficient  use  of  student's  time  during  P.E.  ... 


l 


Greatly  Decreased 


16. 

17. 


How  did  use  of  PLATO  in  practical 
exercises  affect:  (Continued) 


Instructor  availability  to  assist  students 
during  P.E . 

Learning  of  relationships  among  system 
components  and  tests . 
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Please  check  the  one  response  which  best  describes  your  opinion. 

18.  I  would  like  to  see  PLATO  simulations  used  in  other  P.E.  lessons  in  this 
course. 

3  1 )  Strongly  agree 

_  2)  Agree 

_  3)  Undecided 

_  4)  Disagree 

_  5)  Strongly  disagree 

19.  I  would  like  to  see  PLATO  simulations  used  in  otner  courses  at  the  school. 

?  1)  Strongly  agree 

_  2)  Agree 

_____  3)  Undecided 

_  4)  Disagree 

_ _ 5)  Strongly  disagree 

20.  Compared  with  other  training  equipment  I  have  used  (35-mm  projectors, 
video  cassettes,  Seseler  Cue/See),  PlATC's  reliability  is: 

i  1)  Much  better  than  other  training  equipment 

_  2)  Better  than  other  training  equipment 

_  3)  About  the  same  as  other  training  equipment 

_  4)  Worse  than  other  training  equipment 

_  5)  Much  worse  than  other  training  equipment 
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Greatly  Decreased 


21.  In  your  opinion,  how  widespread  is  the  applicability  of  PLATO  simulations 
for  use  in  other  P.E.  lessons  in  this  course? 

3  1)  PLATO  is  applicable  to  alj_  other  P.E.  lessons 

_  2)  PLATO  is  applicable  to  most  other  P.E.  lessons 

_ _  3)  PLATO  is  applicable  to  a  1 imi ted  number  of  other  P.E.  lessons 

_ _  4)  PLATO  is  nut  applicable  to  any  other  P.E.  lessons 

_  5)  PLATO  should  not  have  been  used  in  the  P.E.  1essons  where  it 

was  used 

22.  In  some  courses  there  is  limited  availability  of  equipment  for  training 
purposes.  In  your  opinion,  how  widespread  is  the  applicability  of  PLATO 
simulations  for  teaching  the  operation  or  maintenance  of  equipment  like 
that  taught  at  MMCS? 

?  1)  The  teaching  of  almost  al 1  equipment  items  would  benefit 

by  being  supplemented  by  PLATO  simulations 

_  2)  The  teaching  of  most  equipment  items  would  benefit  by 

being  supplemented  by  PLATO  simulations 

_  3)  A  1 imited  number  of  equipment  items  would  benefit  by 

being  supplemented  by  PLATO  simulations,  but  most  would 
not  benefit 

_ 4)  Current  computer  simulation  technology  is  too  1 imited  to  be 

of  practical  benefit  for  equipment  training  purposes 

23.  If  I  had  the  choice  of  teaching  another  course  where  there  was  major  use 

of  PLATO  simulations  like  those  written  for  this  course,  I  would: 

_ l _  1)  Fight  tooth  and  nail  to  be  assigned  as  the  instructor 

_ [_  2)  Request  assignment,  if  convenient 

_ l_  3)  Neither  request  nor  avoid  assignment 

_  4)  Avoid  assignment,  if  possible 

_  5)  Avoid  assignment  like  the  plague 

24.  Once  students  became  familiar  with  PLATO,  how  much  instructional  assis¬ 
tance  did  a  typical  student  require  when  learning  a  new  task  using  PLATO 
compared  with  using  real  equipment? 

_  1)  Much  more  assistance  needed  when  using  PLATO 

_  2)  More  assistance  needed  when  using  PLATO 

_  3)  About  the  same  amount  of  assistance  needed  using  PLATO 

or  using  real  equipment 

2  4)  Less  assistance  needed  when  using  PLATO 

_ _ l _ 5)  Much  less  assistance  needed  when  using  PLATO 
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In  Questions  25  through  25,  consider  the  Controller  DVM  alignment  test  on  she 
TFTS ,  which  can  be  trained  using  real  equipment  or  a  PLATO  simulation. 

25.  How  fast  could  your  fastest  student  perform  this  task 

on  the  real  equipment?  o.s  and  0.6  hours 

26.  How  fast  could  your  slowest  student  perform  this  task 

on  the  real  equipment  ? .  5  and  1 . 5  hours 

27.  How  fast  could  your  fastest  student  perform  this  task 

on  PLATO?  0.5  ana  0.5  hours 


28.  How  fast  could  your  slowest  student  perform  this  task 

on  PLATO?  1.0  and  1.0  hours 
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TUDENT  EVALUATION  FORMS  AND  RAW  DATA 
END-OF-WEEK  EVALUATION 
END-OF-COURSE  EVALUATION 
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Pita. ie  tell  ua  how  often  you  had  any  o$  the.  pwbleim  lifted  below.  Chech 
one  box  opposite  each  Item  to  Akoui  how  o&ten  the  problem  came  up  In  year 
uAe  o£  PLATO  dunlng  the  week.  PemembeA,  youn.  aniweu  should  only  be  fo-i 
the  week  you  have  just  completed. 


...... 

Never 

Once 

More  Than  Once 

2.  How  often  were  you  unsure  of  what  to  do  next 
(for  example,  touch  the  screen,  hit  a  key, 
or  wait  for  PLATO)?  . 

■ 

14 

14 

3.  How  often  were  you  sure  you  gave  the  right  answer, 
but  PLATO  said  you  were  wrong?  . 

■ 

5 

20 

4.  How  often  did  PLATO  do  something  that  surprised 
you  (for  example,  you  hit  BACK,  but  went 
forward)?  . 

25 

0 

5.  How  often  did  you  hit  a  key  or  touch  the  panel, 

and  nothing  happened?  . 

14 

3 

15 

6.  How  often  was  it  hard  to  study  or  practice  a 

particular  subject  because  you  couldn't  get  to  it, 
or  because  you  had  to  go  through  too  many  other 
steps  to  get  to  the  part  you  wanted?  . 

24 

6 

2 

7.  Hew  often  did  you  have  to  ask  your  instructor  for 
help  because  PLATO  was  not  correct  or  was 
incomplete?  . 

8 

9 

15 

8.  How  often  were  you  confused  because  PLATO  looked 

or  worked  different  from  the  real  equipment?  . 

23 

B 

5 

More  Than  Once 


STUDENT 

END-OF-COURSE  EVALUATION 


Please  complete  this  form,  to  zhe  best  of  your  ability  so  that  this  course  car. 
be  Aullu  and  fair  In  evaluated. 


Course  Name: 
Today's  Date: 


HAWK  C'X  RADAR 


day 


month 


year 


1.  What  did  you  like  about  the  PLATO  part  of  the  course? 


2.  What  did  you  dislike  about  the  PLATO  part  of  the  course? 


3.  On  a  scale  of  1  to  10  (with  10  being  the  highest  score),  how  useful  do 
you  think  the  PLATO  simulations  were? 

Circle  a  number:  1 ‘  2  3^  4"  5^  6°  7 1  8^  9^  10‘ 


4.  Did  the  training  on  PLATO  ever  confuse  you  because  the  real  equipment 
looked  or  worked  differently  than  the  PLATO  simulations? 

12  1 )  No,  it  never  did 

1 3  2)  Yes,  occassionally 
4  3)  Yes,  often 
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5.  When  you  used  PLATO,  how  much  ao  you  tnink  you  learned  comparec  witr, 
using  the  real  equipment? 

_  1)  Learned  much  more  using  PLATO 

4  2)  Learned  more  using  PLATO 

4  3)  Learned  about  tne  same  amount  using  PLATO 

; 7  4)  Learned  less  using  PLATO 

5  5)  Learned  much  less  using  PLATO 

6.  When  you  used  PLATO,  how  fast  do  you  think  you  learned  compared  witr, 
using  the  real  equipment? 

_  1)  Learned  much  faster  using  PLATO 

5  2)  Learned  faster  using  PLATO  i 

i  3)  Learned  in  about  the  same  time  using  PLATO  j 

9  4)  Learned  slower  using  PLATO  ! 

3  5)  Learnea  much  slower  using  PLATO  | 

7.  When  you  used  PLATO,  how  did  it  affect  the  use  of  your  time  during 
P.E. 1 s? 

-1  1 )  Made  much  better  use  of  my  time 

u  2)  Made  better  use  of  my  time  i 

'■j  3)  Made  little  or_  no  difference  in  use  of  my  time 

_ 5_  4)  Made  worse  use  of  my  time 

_ !_  5)  Made  much  worse  use  of  my  time 

8.  How  did  use  of  PLATO  influence  your  confidence  about  operating  the  real  ! 

equipment? 

_  1 )  Greatly  increased  my  confidence 

_ 3_  2)  Increased  my  confidence 

/9  3)  Did  not  change  my  confidence 

_ l_  4)  Decreased  my  confidence 

_  5)  Greatly  decreased  my  confidence 

9.  How  interested  were  you  in  going  through  the  PLATO  lessons? 

_ 12_  1 )  Very  interested  while  using  PLATO 

_ |_5_  2)  Interested  while  using  PLATO 

_ £  3)  Not  interested  while  using  PLATO 

_ £  4)  Bored  while  using  PLATO 

_  5)  Very  bored  while  using  PLATO 
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10.  How  much  of  what  you  learned  using  PLATO  could  be  applied  on  the  job? 
_  1 )  Almost  al 1  of  what  I  learned 

4  2)  Most  of  what  I  learned 
' 5  3)  Some  of  what  I  learned 

1  4)  Little  of  whet  I  learned 

5  5)  None  of  what  I  learned 

11.  For  training  purposes how  adequately  did  the  PLATO  simulations  work  like 
the  real  equipment? 

'  1 )  Very  Adequate  for  training 

12  2)  Reasonably  adequate  for  training 

5  3)  Borderl ine  for  training 

6  4)  Somewhat  inadequate  for  training 

5  5)  Very  inadequate  for  training 

12.  How  much  do  you  think  you  coul d  have  1  earned  about  the  operation  of  the 
real  equipment  using  only  PLATO? 

2  1)  Almost  everything 
_ 5__  2)  A  great  deal 

_ 5_  3)  Some 

1 5  4 )  A  little 

_ 5_  5)  Almost  nothing  at  all 

13.  Compared  with  other  training  equipment  you  have  used  (slide  projector, 
video  cassettes,  Beseler  Cue/See),  PLATO's  reliability  was: 

_ l_  1)  Much  better  than  other  training  equipment 

10  2)  Better  than  other  training  equipment 

12  3)  About  the  same  as  other  training  equipment 

_ 1__  4)  Worse  than  other  training  equipment 

_  5)  Much  worse  than  other  training  equipment 

14.  Would  you  like  to  use  PLATO  for  other  hands-on  training? 

_ 4_  1 )  Would  very  much  1  ike  to  use  PLATO 

_ 6_  2)  Would  1  ike  to  use  PLATO 

_ i_  3)  Undecided 

10  4)  Would  not  1 i ke  to  use  PLATO 
_ l_  5)  Would  very  much  not  like  to  use  PLATO 
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15.  How  would  you  advise  a  friend  wno  nac  a  cnoice  between  taking  a  cojrse 
wnere  there  was  a  great  deal  of  PLATC  simulations  or  anotner  course 
that  covered  the  same  material  but  witnout  PLATO? 

_ l_  1)  Fight  tootn  and  nail  to  get  into  the  PLATO  course 

_ s_  2)  Request  PLATC  course,  if  convenient 

1 }  3)  Neither  request  nor  avoid  course 

]Q  4)  Avoid  PLATO  course,  if  possible 

_ l_  5)  Avoid  °LAT0  course  like  the  plague 

16.  How  much  time  did  you  spend  on  PLATO  compared  with  other  students  in  your 
class? 

_ !_  1 )  More  time  on  PLATO  than  ctners 

ZS  2)  Aoout  the  same  amount  of  time  on  PLATO  as  others 
_ £  3)  Less  time  on  PLATO  than  others 


PLATO  INSTRUCTOR/STAFF  QUESTIONNAIRE 

Please  complete  this  form  to  the  best  cf  your  ability  so  that  this  course  car. 
be  fully  and  fairly  evaluated. 

Course  Name:  _ HAWK _ 

Today's  Date:  _  _  _ 

day  month  year 


1.  Please  check  the  item  which  best  describes  your  role  in  tnis  course. 

_J_ _  1^  Regular  instructor  (Answer  all  questions.) 

_ 2)  School  staff  (Stop  after  Question  5.) 

Specify  position:  _ 

2.  What  do  you  feel  are  the  instructional  advantages  of  using  PLATO  in  this 

particular  course?  _ 


3.  What  do  you  feel  are  the  instructional  disadvantages  of  using  PLATO  in 
tnis  particular  course?  _ 


4.  What  changes  in  training  practices  occurred  in  this  particular  course 
because  of  the  use  of  PLATO? 


5.  If  the  use  of  PLATO  became  more  widespread  (more  lessons  in  this 
course  and/or  more  courses),  how  would  your  role  be  changed?  _ 


* 


- r - 

In  Questions  6  through  Z?3  cotKpcre  oractical  exercises  using  PLATO  with 
practical  exercises  using  only  real  equipment.  For  each  question ,  please 
check  the  one  box  which  best  describes  your  opinion  about  the  impact  of  PLATO 
simulations  in  this  course. 


How  did  use  of  PLATO  in  practical 
exercises  affect: 

Greatly  Increased 

Increased 

little  or  No  Chan 

Decreased 

Greatly  Decreased 

6. 

Student  learning  during  P.E . 

4 

l 

7. 

Instructor 1 s'conf idence  and  feeling  of  use¬ 
fulness  during  P.E . 

■ 

2 

2 

i 

1 

i 

8. 

Student  attention  during  P.E . 

■ 

3 

1 

1 

1 

9. 

Student  confidence  when  dealing  with  real 
equipment.  . 

■ 

■ 

5 

I 

/ 

10. 

Use  of  TM's . 

■ 

3 

1 

11. 

Instructor's  workload  durino  P.E . 

■ 

| 

1 

12. 

Application  of  learning  to  job  performance.  ... 

■ 

3 

H 

■ 

13. 

Damage  to  equipment  items  used  for  training.... 

14. 

Student  overconfidence  when  dealing  with 
real  equipment . 

■ 

■ 

3 

15. 

Efficient  use  of  student's  time  during  P.E.  ... 

■ 

■ 

B 
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Greatly  Decreased 


How  did  use  of  PLATO  in  practical 
exercises  affect:  (Continued) 
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16.  Instructor  availability  to  assist  students 
during  P.E . 


17.  Learning  of  relationships  among  system  ;  J 

components  and  tests .  !  |  !  3 


Please  check  the  one  response  which  best  describes  your  opinion. 

18.  I  would  like  to  see  PLATO  simulations  used  in  other  P.E.  lessons  in  this 
course. 

_  1)  Strongly  agree 

_  2)  Agree 

_  3)  Undecided 

_ 5_  4)  Disagree 

_ j_  5)  Strongly  disagree 

19.  I  would  like  to  see  PLATO  simulations  used  in  other  courses  at  the  school. 
_  1)  Strongly  agree 

_  2)  Agree 

_ 3_  3)  Undecided 

_  4)  Disagree 

_ l_  5)  Strongly  disagree 

20.  Compared  with  other  training  equipment  I  have  used  (35-mm  projectors, 
video  cassettes,  Seseler  Cue/See),  PLATO'S  reliability  is: 

_  1)  Much  better  than  other  training  equipment 

_ l_  2)  Better  than  other  training  equipment 

_ l_  3)  About  the  same  as  other  training  equipment 

_  4)  Worse  than  other  training  equipment 

_  5)  Much  worse  than  other  training  equipment 
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Greatly  Decreased 


21.  In  your  opinion,  how  widespread  is  the  applicability  of  PLATO  simulations 
for  use  in  other  P.E.  lessons  in  this  course? 

_  1)  PLATO  is  applicable  to  al_l_  other  P.E.  lessons 

_  2)  PLATO  is  applicable  to  most  other  P.E.  lessons 

_  3)  PLATO  is  applicable  to  a  1 imited  number  of  other  P.E.  lessons 

3  4)  PLATO  is  not  applicable  to  any  other  P.E.  lessons 

2  5)  PLATO  should  not  have  been  used  in  the  P.E.  lessons  where  it 

was  used 

22.  In  some  courses  there  is  limited  availability  of  equipment  for  training 
purposes.  In  your  opinion,  how  widespread  is  the  applicability  of  PLATO 
simulations  for  teaching  the  operation  or  maintenance  of  equipment  like 
that  taught  at  MMCS? 

_ _  1)  The  teaching  of  almost  all  equipment  items  would  benefit 

by  being  supplemented  by  PLATO  simulations 

_ _ 2)  The  teaching  of  most  equipment  items  would  benefit  by 

being  supplemented  by  PLATO  simulations 

3  3)  A  limited  number  of  equipment  items  would  benefit  by 

being  supplemented  by  PLATO  simulations,  but  most  would 
not  benefit 

_ 1_  4)  Current  computer  simulation  technology  is  too  limited  to  be 

of  practical  benefit  for  equipment  training  purposes 

23.  If  I  had  the  choice  of  teaching  another  course  where  there  was  major  use 
of  PLATO  simulations  like  those  written  for  this  course,  I  would: 

_  1)  Fight  tooth  and  nail  to  be  assigned  as  the  instructor 

_  2)  Request  assignment,  if  convenient 

_ 2_  3)  Neither  request  nor  avoid  assignment 

i  4)  Avoid  assignment,  if  possible 
_  5)  Avoid  assignment  like  the  plague 

24.  Once  students  became  familiar  with  PLATO,  how  much  instructional  assis¬ 
tance  did  a  typical  student  require  when  learning  a  new  task  using  PLATO 
compared  with  using  real  equipment? 

_  1)  Much  more  assistance  needed  when  using  PLATO 

_ 3_  2)  More  assistance  needed  when  using  PLATO 

_ I_  3)  About  the  same  amount  of  assistance  needed  using  PLATO 

or  using  real  equipment 

_  4)  Less  assistance  needed  when  using  PLATO 

_  5)  Much  less  assistance  needed  when  using  PLATO 


In  Questions  25  through  28,  consider  the  Controller  DVM  alignment  test  on  the 
TTTS,  which  can  be  trained  using  real  equipment  or  a  PLATO  simulation. 


25. 

How  fast  could  your  fastest 
on  the  real  equipment? 

student  perform  this 

task 

N/A 

hours 

26. 

How  fast  cculd  your  slowest 
on  the  real  equipment?  ~ 

student  perform  this 

task 

N/A 

hours 

27. 

How  fast  could  your  fastest 
on  PLATO? 

student  perform  this 

task 

N/A 

_ hours 

28.  How  fast  could  your  slowest  student  perform  this  task 

on  PLATO?  N/A  hours 


